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Preface 


Th(S  report  is  a collection  of  three  papers  that  describe  the  design  and  implementation 
of  the  multi-microprocessor  computer  system  (Cm»)  currently  under  developement  at 
Carnegie  Mellon  University.  The  three  papers  are: 

Cm*:  A Modular  Multiprocessor 

The  architecture  of  a new  multiprocessor  that  supports  a large  number  of 
processors  (on  the  order  of  100)  is  described  in  this  paper.  The  system  enables 
very  close  cooperation  between  the  large  numbers  of  inexpensive  processors.  All 
processors  share  access  to  a single  virtual  memory  address  space.  There  are  no 
arbitrary  limits  on  the  number  of  processors,  amount  of  memory  or  communication 
bandwidth  in  the  system.  Considerable  support  is  provided  for  low  level 
operating  system  primitives  and  inter-process  communication 

The  Implementation  of  the  Cm*  Multi-Mici  oproccssor 

The  implementation  of  the  Cm*  multiprocessor  multiprocessor  is  presented.  The 
lowest  level  of  the  structure,  a Computer  Module,  is  a processor-memory  pair 
Computer  Kiodu'ies  are  grouped  to  form  a cluster;  communication  within  the 
cluster  is  via  a parallel  bus  controlled  by  a centralized  address  mapping 
processor.  Clusters  communicate  via  intercluster  busses.  A memory  reference  by 
a program  may  be  routed,  transparently,  to  any  memory  in  the  system.  This 
paper  discusses  the  hardware  used  to  implement  the  communication  mechanism. 
The  use  of  special  diagnostic  hardware  and  performance  models  is  also  discussed 

Software  Management  of  Cm*,  a Distributed  Multiprocessor 

This  paper  describes  the  software  system  being  developed  for  Cm*,  a 
distributed  multi-microprocessor.  This  software  provides  for  flexible,  yet 
controlled,  sharing  of  code  and  data  via  a capaoility  addressed  virtual  memory, 
creation  and  management  of  groups  of  processes  Known  as  task  forces,  and 
efficient  interprocess  comniunicalion. 
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Cm>h:  a Modular,  Multi-Microprocessor 


R J.  Swan 
S H Fuller 
0.  P Slewiorck 


Carnegio-Mellon  University 
Pittsburgh,  PA  15213 


Cm*  IS  an  experimental  computer  system  designed  to 
investigate  the  problems  and  potentials  of  modular,  multi- 
microprocessors. The  initial  impetus  for  the  Cm"  project 
was  provided  by  the  continuing  advances  In  semiconductor 
technology  as  exemplified  by  processora-on-a-chip  and 
large  memory  arroys  In  the  near  future  processors  of 
moderate  capability,  such  as  a PDP-11,  and  several 
thousand  words  of  memory  will  bo  placed  on  a single 
Integrated  circuit  chip  If  largo  computer  systems  are  to  be 
built  from  such  chips,  what  should  bo  the  structure  of  such 
a 'computer  module'? 


Novonibor  30,  1976 


Abstract 

This  paper  describes  the  architecture  of  a new  large 
imiltiprocessor  computer  system  being  built  at  Carnegie- 
Mellon  University  The  system  ollov/s  close  cooperation 
between  large  numbers  of  Inexpensive  processors.  All 
processors  share  access  to  a single  virtual  memory  address 
spncc.  There  are  no  arbitrary  limits  on  the  number  of 
processors,  amount  of  memory  or  communication  bandwidth 
111  the  system.  Considerable  support  is  provided  tor  low 
level  operating  system  primitives  and  inter-process 
communication  . 


Cn  Cnteyortos  0.20,  4.30,  4.32 

Keywords:  Mjitiprocessor.  microprocossor, 

nrclutocture,  virtual  memo,  v 


Initial  versions  of  the  Cm“  architecture  [Fuller,  et  m 73] 
grew  In  part  as  an  extension  to  the  modular  design  of 
systems  from  register  transfer  modules,  or  RTMs  [Dell  et  al, 
72]  In  odd.'tior)  there  was  substantial  interest  In  the 
dcvrlOf>mt;nt  of  large  multiprocessor  systems  such  as 
Plurtbii^  [heart,  et  al  73]  and  C.mmp[WuIf  and  Bell,  72] 
Cm*  Is  intended  to  be  o testbed  for  exploring  a number  of 
research  questions  coi^cernmg  multiprocessor  systems,  for 
example  potential  for  deadlocks,  structure  of  inter- 
processor  control  mechanisms,  modularity,  reliability,  and 
techniques  for  dccor^posmg  algoritlims  into  parallol 
cooperating  processes 

The  structure  of  Cm*  is  very  briefly  described  in  Section 
2.  Section  3 Is  a description  of  the  address  structure  and 
rtrscpsses  the  support  tj>vpn  for  flip  opproting  system  The 
use  of  the  addressing  structure  for  Inter-process 
communication  and  control  operations  is  discussed  in 
Section  4.  A companion  papor  [Swan,  et  al.  77]  d.scusses 
the  vorious  mechanisms  used  to  implement  the  complex 
address  mapping  and  routing  structure  of  Cm*.  Some 
results  from  the  perfornonco  modeli'ng  of  Cm*  are  also 
presented  A second  companion  paper  [Jones,  et  at  7 7] 
describes  the  structure  ot  the  basic  operating  .system  and 
support  software. 


computer 


2.  The  Structure  of  Cm* 


This  work  was  supported  in  part  by  the  Advanced 
Research  Projects  Agency  under  contract  F44620-73-C- 
00  74.  which  IS  monitored  by  the  Air  Force  Office  of 
Soent'fle  Research,  and  in  part  by  the  National  Science 
Foundation  Grant  GJ  32768X  The  LSI-1  Vs  and  related 
equipnu'nt  were  supp'iod  by  Digital  Equipment  Corporation. 


There  is  a surprising  divor.sily  of  ways  to  apprr'arh  tiM' 
interconnection  of  proce.ssors  into  a computinq  systeni 
[Anderson  and  Jensen.  7B]  The  processors  could  hi‘ 
interconnect od  with  sc’vorni  .serial  I/O  links  tc  form  a 
computer  network,  they  could  be  interconnected  In  a t’qht 
syrichronous  foshion  to  h nid  an  army  processor,  or  the 
processorr.  could  he  crg.intzocJ  to  snnro  primary  memory 
This  lost  approach,  a multiprocessor  organ'zotion.  was 
chosen  for  Cm*  because  it  offers  a closer  degrpo  o* 
coupi'iiq.  or  cornfruMicotii'n.  between  the  processors  thfln 
would  a muUicomputor  or  netwc'K  configur.ilion 
Miiltiproci»ssorn  also  h.ive  a mere  general  range  <)f 
applicobiiity  than  otho'  mu  t-pie  procesr.er  systems 
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Figure  2.1  Canonical  Computer  Modulo  Structure 


During  the  development  of  the  Cm*  structure  a wide 
var«oty  of  multiprocessor  switch  structures  were  considered 
[Swan,  ot  al.  7G0].  The  basic  structure  selected  Is 
represented  in  Figure  2.1.  The  essential  feature  which 
distinguishes  It  from  other  multiprocessor  structures  is  that 
shared  memory  is  not  separated  from  the  processing 
elements,  hut  rather  a unit  of  memory  and  a processor  are 
closely  coupled  m each  modulo  and  a network  of  buses 
gives  a processor  access  to  nonlocal  memory  This 
structure  allows  modular  expansion  of  the  number  of 
processors  and  memory  modules  witlio\ii  o rapid  increase  in 
the  interconnection  costs.  Memory  can  be  shared  even 
though  there  is  no  direct  physical  connection  between  the 
regvicsting  processor  and  the  recjuired  memory.  For 
example,  consider  n request  by  processor,  PI,  to  the 
momory,  M4,  in  Figure  2.1.  The  address  mapping  element, 
K1,  directs  the  reference  from  PI  onto  the  intermodi/lo  bus. 
The  nddrnjjs  i.s  recognized  by  K2.  which  directs  it  onto  a 
second  inter-module  bus.  The  reference  is  finally  accepted 
by  K4,  which  accesses  tho  request  memory  location  and 
passes  back  an  acknov^lodgement  or  data  to  the  reejupsting 
processor.  The  need  for  higli  inter -module  communication 
rates  will  be  minimized  if  a largo  fraction  of  each 
processor’s  references  to  primary  memory  'hit'  the  section 
of  m<>mory  local  to  the  processor  (Prolunmary  experiments 
In  the  Fall  of  1576  indicate  that  h.t  ratios  of  bettor  than 
90%  can  be  expected  provided  that  the  code  executed  is 
normally  hold  locol  to  the  processor  ) 


2.1  Deadlock  with  References  to  Nonlocal  Memory 

Almost  ail  computer  systems  Implement  accesses  from 
processor  to  primary  memory  with  Circuit  Switching,  that 
IS.  a complete  path  is  established  from  a processor  to  the 
memory  being  referenced  Circuit  switching  Is  not  feasible  i 

for  a structure  like  Cm*  where  local  memory  Is  also  I 

accessible  as  shared  memory  Figure  2.1  shows  tlie  path  ; 

used  for  P1  to  access  M4  via  K2  Consider  a concurrent  ’■ 

attempt  by  P4  to  access  Ml  via  K2  With  a circuit  switch  i 

implomentation,  a situation  could  arise  where  P1  held  Its 
focal  memory  bus  and  the  bus  connecting  K2,  while  P4  also 
holds  its  own  memory  bus  plus  the  bus  connecting  K4  to  K2. 

Ncither  memory  reference  could  complete  without  one 
processor  first  releasing  the  buses  it  holds  There  are 
numerous  situations  where  deadlock  over  bus  allocotion  can 
occur  Resolving  this  deadlock  requires,  at  the  very  least,  a 
timeout  and  retry  mechanism. 

The  alternative  to  circuit  switching  is  PacMet  Switching 
In  a packet  switched  implomentation,  the  address  from  tho 
processor  is  latched  at  each  level  in  the  bus  structure 
Buses  are  not  nllocniod  for  tho  full  duration  of  a memory 
reference,  but  just  for  the  time  taken  to  pass  a ‘packet*, 
containing  an  address  and/or  data,  from  one  node  on  the 
bus  to  another.  Therefore  packet  switching  allows 
significantly  better  bus  utilization  and  significantly  reduced 
bu.s  contention  in  Cm*-like  structures.  The  use  of  packet 
switcbi/>g  Dh/vin.itoa  the  possibihty  of  deadlock  over  bus 
allocotion  but  introduces  the  possibility  of  deadlock  over 
buffer  allocation.  [Fuller,  et  al  73.  Swan,  et  al.  7CA] 

Duffers,  or  intermodiatc  registers,  are  resources  which  con  , 

be  provided  very  cheaply,  relative  to  providing  additional 
inter-Cm  buses,  with  present  technology. 


2.2  The  Actual  5*1  ..''♦ure  of  Cm* 

Design  studios  indicated  that  very  little  pp'formoncn 
loss  would  result  fro  coml'ining  several  individual  Compute'’ 
Modules  into  n cluster  and  provid*ng  a shared  a Jdn*ss 
mapping  and  routing  processor.  Kmap,  which  allowed 
communication  with  other  cKistors.  Because  tiio  cost  of  tho 
Kmap  IS  distributed  across  man/  processors  it  can  bi» 
eridov/od  wfth  consideraule  flexM)ii<ty  and  power  at 
relatively  httU*  Incremontal  cost  Because  cf  its 

commanding  position  in  the  cluster,  the  Kmap  con  onsure 
mutual  exclusion  on  access  to  shared  data  structures  with 
very  little  overhead 

The  full  structure  of  Is  shown  in  Figure  ? p 

Indivitliial  Computer  Motlulos.  or  Cns.  consist  of  a ('FT  ISi- 
11  processor,  nn  Siocnl  and  standa'd  lM-11  bus  mempfv 
nnrj  devices  Pie  p'oeessor  is  prournm  comp.itiMo 
POP- 11s.  thus  a Iftr  JO  liocly  of  sn'twaro  is  immedMlriy 
ovnilahh'  Iho  prm  o function  of  tho  S ccnl,  or  local  switch,  is 
to  direct  references  from  the  prov.es.sor  se'or.tively  edher 
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Intercluster  Bus 


P-S-M  P-S-M  P-S-M 

A Cluster  of  Computer  Modules 


u 


Figure  2.2  A Simple  3 Cluster  Cm*  System 

to  local  memory  or  to  the  Map  Bus,  and  to  accept 
references  from  the  Map  Bus  to  the  local  memory. 

Up  to  14  Computer  Modules  and  one  Kmap  form  n '"luster.  3.  Architecture  of  the  Address  Translation  Mechanisms 
The  Kmop,  or  mapping  processor,  consists  of  three  major 

components.  The  Khus  arhitrntes  and  controls  the  Map  bus.  Many  of  the  n'oro  conventional  aspects  of  the 

The  Pmap  Is  a horizontally  microcoded  150  ns  cycle  time  architecture  of  the  Cm"  system  ere  consequences  of  using 

processor.  The  basic  configuration  has  1 K x 80  bits  of  LSI-1  1 *s  for  the  central  p.'oceiSing  elements.  The 

writable  control  store  and  5K  x 1 6 bits  of  bipolar  BAM  tor  organization  and  encoding  of  the  in.sti  ctions,  Interrupt  and 

holding  mapping  tobies  etc.  The  third  level  of  the  Cm*  trap  sequencing,  and  the  64K  byte  processor  address 

structure  is  provided  by  the  Interclustof  buses  which  allow  spa^  ^ of  a Cm"  system  are  all  a result  of  the  POP-1  1 

communication  between  clusters.  The  Line  provides  the  architecture  as  Implemented  on  the  LSi-1  1 By  selection  of 

interface  to  two  mlercliister  buses.  the  LSI-1  1.  hov^over.  wo  do  not  want  to  imply  that  the  PPP- 

1 1 architecture  is  idonhy  suited  to  multiprocessor  systerns 
there  are  no  arbitrary  limits  to  the  size  of  a Cm"  system.  The  ideal  solution  would  have  been  for  us  to  have  dos  giind 

Mc"iorlc3.  processors  and  Kmaps  can  be  Incromentol'y  our  own  processors.  Hovrovor,  practical  considerations  rd 

added  to  suit  needs  Any  processor  can  access  any  memory  time,  money,  and  existing  support  .software  lead  us  in  eaiK 

locaticn  in  the  system  The  routing  of  a processor's  1075  to  recoijmrn  that  by  chosmg  the  lSI-11  we  could 

reference  to  a target  memory  is  trnnsporent  to  the  program,  concentrate  on  those  aspects  of  the  Cm"  architecture 

thus  the  system  can  be  reconfigured  dynamically  in  unique  to  multiprocessor  systems  Tins  section,  and  tlin 

response  to  hardware  failures  following  section  on  control  structures  will  rnscuss  the  Cm" 

architecture  ns  wo  extonuod  it  licyond  the  standard  PDP- 
1 1 architecture 

The  add-r'SSi.ig  .strurtnto  is  one  of  the  the  most 
important  aspects  of  any  co.mputnr  architecture,  it  is  even 
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more  significant  when  cooperation  between  multiple 
processors  is  to  be  achieved  by  sharing  an  address  space. 

Denning  [1970]  lists  four  objectives  tor  a memory  mapping 
scheme: 

(a)  Program  modularity  the  ability  to  independently 
change  and  recompile  program  modules 

(b)  Variable  size  data  structures. 

(c)  Protection 

(d)  Data  and  program  sharing  allowing  independent 
programs  to  access  the  same  physical  memory 
addresses  with  different  program  names. 

For  Cm*,  where  we  are  using  processors  with  only  a 64K 
byte  address  space,  we  must  add  the  following 
requirement: 

(0)  Expansion  of  a processor's  address  space. 

Processor  generated  Physical  Address 

Cm"  has  a 2^®  byte  segmented  virtual  address  space.  Address  on  LSI-11  Bus 

Seg.'ntints  are  of  variable  size  up  to  a maximum  of  4K  bytes. 

There  is  a capability-based  protection  scheme  enforced  by  Figure  3.1 

the  Kmap.  The  odtircssing  structure  provides  considerable  Addressing  fVlechanisnn  for  Local  Memory  References 

support  tor  operating  system  primitives  such  as  context 

switching  and  mtorjiroccss  mossago  transmission.  Modulo  is  ni  t involved.  When  the  relorcnce  is  complete  the 

Kbiis  trnnsters  the  data  read  from  the  destination  Slocal 
directly  bTk  to  the  requesting  processor  via  the  Map  bus 
anil  Its  SiouQi. 

3.1  Tiie  Path  from  Processor  to  Memory 

If  the  processor  references  a segment  (n  another 
The  Slocal  (see  Figures  2 2 and  3 1)  provides  the  first  cluster  then  the  Pmap  will  transmit  a request  to  the  desired 

level  of  memory  mapping.  A reference  to  local  memory  is  cluster  via  the  Line  end  the  Interclustcr  buses.  (See  fig 

simply  relocated,  on  4K  byte  page  boundaries,  by  the  2 2.)  If  the  destination  cluster  is  not  directly  connected  to 

relocation  table  in  the  Slocal  As  discussed  above,  it  is  fho  source  cluster,  tfiat  Is.  If  it  docs  not  share  a common 

ossumr-d  that  most  memory  references  will  be  made  by  Intcrckister  bus,  then  the  message  will  be  automatically 

processors  to  their  local  memory  Relocation  of  local  routed  via  Interricdiate  clusters  When  the  message 

memory  references  can  be  implemented  with  no  reaches  the  destination  cluster,  the  memory  reference  is 

performance  overhead  because  the  synchronous  processor  performed  similar  to  a request  from  a processor  within  the 

has  sufficiently  wide  timing  margins  at  the  points  where  cluster.  An  ackno'wiedgemcnt,  or  Return,  me.ssago 

address  relocation  is  performed  For  segments  which  are  (containing  data  in  the  case  of  a read)  is  always  sent  back 

not  in  a processor’s  local  momory  the  relocation  table  has  a fo  the  source  cluster  and  subsequently  to  the  requesting 

status  bu  which  causes  the  addre.ss  to  be  latched,  the  processor 

processor  forced  off  the  LSI-1  1 bus,  and  a Service  Rerpiest 
to  he  signalled  the  Kmop.  Ail  transactions  on  the  Map  bus 
ore  contrnMod  by  tlio  Map  bus  controller,  or  Kbus,  which  Is  a 

comjionent  of  the  Kmr.P.  The  adrjrrss  generated  by  the  3.2  The  Addressing  Environmcht  of  a Process 
processor  is  transferred  via  the  Man  bus  to  the  Pmap,  the 

mirroproiirnmmod  processor  within  the  Kmap.  If  the  The  vuliial  address  space  of  Cm"  is  subdivided  into  up 

reference  is  for  memory  within  the  cluster  then  the  Pmap  to  2^*^  Segments  Each  segment  is  defined  by  a Scs;mcnl 

generates  a physical  address  and  sends  It  to  the  Descriptor  The  stand, \rrt  tyiio  of  segment  is  smulnr  to 

appropriate  Slocol  If  it  is  a write  operation,  data  is  passed  segments  m other  computer  systems,  it  ,s  smiplv  « vector  of 

directly  from  the  source  Slocal  to  the  destination  Slocal;  momory  locations  The  se.jment  doscript.nr  specifies  the 

the  data  docs  not  have  to  bo  routed  through  the  Kmao  The  physical  base  address  of  the  segment  and  the  length  of 

selected  destination  Slocal  performs  the  requested  memory  the  segment.  Segments  are  variable  in  size  from  2 bytes  to 

reference  and  the  processor  In  the  destination  Computer  4 K,  bytes  Ho-wever,  cither  segment  ty(.-es  may  he  more 
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thun  simple  linear  vectors  of  memory,  references  to 
segments  may  invoke  special  operations.  Segments  may 
have  the  properties  of  stocks,  queues  or  other  data 
structures.  Some  segments  may  not  have  any  memory 
associated  with  them,  and  a reference  to  the  segment 
would  invoke  a control  operation  For  each  segment  type, 
up  to  eight  distinct  operations  can  be  defined.  For  normal 
segments  the  operations  are  Read  and  Write  Conceptually, 
segments  are  never  addressed  directly;  they  are  always 
referenced  indirectly  via  a CapabiUty.  A capability  is  a two- 
word  Item  containing  the  name  of  a segment  and  a Rights 
field.  Each  bit  in  the  rights  field  Indicates  whether  the 
corrosponding  operation  is  permitted  on  the  segment. 


Capability  List,  CL[0].  The  first  entry  In  CL[0]  Is  a 
Capability  for  a Stare  i/cctor,  which  holds  the  process  stote 
whifo  It  13  not  executing  on  a processor  Entries  CL[0](1) 
to  CL[0](7)  in  the  Primary  Capability  list  may  contain 
Capabilities  for  SeconOory  Capability  Lists  referred  to  as 
CL[1]  through  CL[7]  respectively  The  remaining  entries  in 
the  Primary  Capability  List  and  all  the  entries  In  the 
Secondary  Capability  Lists  contain  Capabilities  for  segments 
which  can  be  mode  directly  addressable  by  the  process 
when  It  executes.  These  may  be  code,  data  or  any  other 
type  of  segment.  The  provision  of  up  to  eight  Capability 
Lists  facilitates  the  sharing  of  segments  and  sets  of 
segments  by  cooperating  processes  A software  module 
con  only  acce.ss  those  segments  for  which  it  has 
capabilities  and  perform  only  those  operations  permitted  by 
the  capabilities. 


3.3  Virtual  Address  Generation 


20 

2 Byte  Virtual 
Address  Space 


Figure  3.3 

Windows  from  tlic  Processor's  Immediate  Address  Space 
to  the  Virtual  Address  Space 


Figure  3.2  Tlic  Enuirenment  of  a User  Software  Modulo 

to  prouiclp  pffi.riont  support  for  context  swnopiny. 
messnrjo-sontliurj  etc  . it  is  necossarv  for  the  Kniap 
niicrocoOe  to  umferstnnrl  some  ot  the  structure  of  «n 
oxecutnblo  software  module  (variously  called  a process, 
activity,  address  space  etc  ) Each  oxocutahle  software 
modulo  IS  roptosentod  hy  an  irwnro'trnent.  Titjure  't  ? An 
oriviionmGnt  IS  a throo-lovel  structure  composed  of 
serimonts  The  first  leve'  Irt  the  structure  is  a Pnnary 


the  processors  ih  Cm*.  iSI-1*s.  can  directly  peneralo 
only  a 1fi  '.id  adonrss.  tins  6d  K hytu  a.Idress  spare  is 
divaled  into  16  par|us  of  A K bytes  each  Lach  page 
provides  a window  nto  the  system  wide*  byte  vi'Iual 
atldross  sp.ice.  (see  f i.'jur,.  2 0)  and  can  he  inrlepenit.'i'th. 
boiiiul  to  a different  se.ymi'nt  i.n  the  virtual  nddnrss  sp.tce 
The  top  p.tiie  in  the  prpcessor's  address  spare  p.i'ie  16  s 
reserved  for  direct  preyrani  interaction  w.tli  tlie  ^mop  Tii  s 
niechan.sn  is  ana.otious  to  the  I/O  pnge  conventi-m  m 
starnlii'd  i’0l'’-1is.  in  pnuo  15  there  are  15  p-.i-iido 


a 
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registers,  called  tV/nc?oiv  Registers  Tliese  define  the 
binding  between  page  frames  in  the  processor's  immediate 
address  space  and  segments  in  the  virtual  address  space. 
This  binding  is  done  Indirectly  via  capabilities  Each  window 
register  holds  an  index  for  a capability  In  the  currently 
executing  software  modulo's  capability  list  structure  A 
Capability  List  index  consists  of  a three  bit  field  to  select 
one  of  the  up  to  eight  Capability  Lists,  plus  an  offset  within 
the  C-List. 

To  overlay  the  processor's  address  space,  ie.  to  change 
the  mapping  from  page  frame  to  segment  m the  virttial 
address  space,  a program  simply  writes  a new  copability 
Index  Into  tho  appropriate  window  register,  This  overlay 
operation  »s  completely  protected,  the  program  con  only 
reference  segments  for  wluch  it  has  o Capability  The  act 
of  writing  tlie  Capability  index  into  the  window  register 
activates  the  Kniop.  The  Kmop  retrieves  tlie  selected 
Capability  from  main  memory  and  places  it  in  its  "Capability 
cacite".  The  Kmap  adjusts  its  internal  tables  so  that 
subserjuent  references  to  tlie  page  frame  will  map  to  the 
segment  specified  by  the  Capability.  If  the  segment  Is  local 
to  the  processor  then  the  Kmap  may  also  change  the 
relocation  register  in  the  Slocnl  so  that  references  to  the 
segment  can  be  porformod  at  full  speed  withou;  the 
intervention  of  the  Kmap,  The  SIocqI,  for  cost  and 
porforniance  reasons,  does  not  have  the  hardware 
necessary  for  bounds  checking  on  variable  sized  segnients. 
Thus  only  fixed  s.ze  4 K byte  segments  can  be  accessed 
wltlv>ut  Kmap  assistance. 

T)»e  Cm*  mechanism  for  address  space  overlaying  should 
bo  contrasted  witli  mechanisms  in  other  computer  systems. 
When  executing  a la^’pe  program  on  a processor  witli  a small 
immediate  address  space,  the  tune  taxen  to  overlay  the 
address  space  can  have  a crucial  effect  on  performance 
NVjasurements  made  of  the  execution  of  the  operating 
system  HYDRA  [Wulf  et  al.  76]  on  the  C.mmp  multiprocessor 
showed  that  relocation  registers  were  being  changed 
a.pproxmiatcly  every  12  instructions  (This  docs  not, 
however,  imply  that  user  programs  perform  overlay 
operations  this  frequently.)  Within  tho  operating  system 
this  overlay'  oi)cration  is  a smgio  PDP-11  MOVE  instruction 
l)ocsuso  no  protection  is  involved.  However  for  user 
(>rograms  running  under  HYDRA,  an  overlay  operation 
re(}Uires  invocation  of  the  operating  syste.m  with  several 
fuindrod  histruct.ons  of  softw<^re  over)ii*arJ.  Subsequent 
optimization,  and  partial  microcoding,  have  greatly  reduced 
this  overhead. 

Figure  3 4 shows  tho  coiiccptual  translation  from  a 1G 
bit  procossor-generntod  address  to  a virtual  address  The 
four  higi»  order  address  bits  from  the  nrocessor  select  one 
of  1G  Wmdov/  registe.'s.  The  Window  register  holds  an 
index  for  a Capability  In  tho  executing  software  modules 
Cfipribitity  List  structure  Tho  16  b't  segment  name  from  tho 
selected  Caoab.hty  is  concatenated  w.tn  the  1?  low  ordof 
bits  from  the  processor  to  form  a 26  b t vuiua!  address 


Cm*-  a Modular,  Multi-Microprocessor 


Window  Register  Capability 


16  Bit,  Processor  28  Bit.  System  Wide 

Generated  Address  Virtual  Address 


Figure  3.4 

Conceptual  Virtual  Address  Generation  and  Rights  Checking 

Figure  3.4  also  shows  tlie  rcad/wnfe  indicator  from  tiie 
processor  being  concatenated  with  t-wo  bits  in  the  addrrjss 
expansion  registers  to  form  a throe  bit  opcode  Tlie 
corresponding  bit  In  tim  Capability  rights  field  is  selected 
and  tested.  If  the  operation  Is  not  permitted  then  an  error 
trap  is  forced. 


3.4  Virtual  to  Physical  Address  Mapping 

The  mapping  from  vuluul  to  physical  address  depends  o-i 
the  location  of  th.o  s'^gment  m the  network  and.  of  coc;r*e. 
on  tlie  type  of  tlie  segment  We  beejm  wdh  tho  case  of  n 
simpio  read/wrilo  segmont  residing  withm  the  sanT  cUr>t«'f 
as  tlie  processor  referencing  tiie  segment.  This  nijop-tr’  •*> 
shown  in  Figure  3 5 Tho  segment  name  is  used  to  access 
tliG  corresponding  seg.ment  descriptor  The  dosc''i''’ 
provides  a limit  vaiuc  winch  is  chockod  aqHinst  tn«'  1.  b t 
offset  in  the  v.rtuoi  address,  if  the  roforunco  is  out  of  ti  n 
bounds  of  Hie  segmont  then  an  error  trap  occurs  Tne 
offset  is  adficd  to  the  physical  base  anclress  from  th(> 
descriptor.  The  n. suiting  16  bit  value  fs  a physiCvS.  idures'. 
witliin  the  256  K byte  nddrer.s  space  of  tho  corrut*  ’ 
module  also  specified  m the  ooscriptor 

If  the  vtfluai  addro:>.«.  referencu-j  a soq'"€‘nt  out*  ><10  t'n* 
source  cluster  then  the  seqrr.u'nt  ramo  u'^erl  to  ni;ce*.s  «n 
Indirect  D<:scnptot  Reicrcnce  ratfic'  iha»i  tlie  uosc'iHor 
Itself  The  indirect  re'erenco  s mpiv  i.idicstes  wn- m 
cluster  the  segment  fcs'des  The  K-nnp  then  passes  tin- 
virlurtl  ndftress  to  that  cluster  via  t--.p  inter- custtu  luis*  . 
An  nlterndtive  approuen  or  to  have  dupi'cat*’  • r»  . 

of  Hic  segment  dos-'.rii-vtnf.s  in  i»ve'y  Custer  t’v..  t‘ 
virUiiil-to-phvsiCrti  .nanpiMj  COu  d be  done  at  •*  e s»  o»rM 
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Segment  Descriptor 

Type  Limit  Base  Address 


Figure  3.5  Virtual  to  Piiysical  Address  Mapping 
for  a Variable  Sized  Segment 

clustor.  With  possibly  some  savi.igs  m overhead  Mo.vnvcr, 
any  attempt  to  change  the  vh t»iul-to-physical  binding  of  a 
segment  (c  g.  moving  it  to  a different  memory  module  or  onto 
backing  store)  would  roguire  an  effectively  simultaneous 
change  to  all  copies  of  the  segment  descriptor.  In  a large 
network  this  operation  would  be  slow  and  cumbersome,  if 
not  impossible.  A further  advantage  to  ensuniuj  that  only  o 
single  descriptor  exists  for  each  segment  is  that  a Lock  Bti 
can  oo  provuiQci  in  the  descriptor  Tiie  lock  bit  can  be  used 
to  ensure  mutual  exclusion  for  special  segment  operations. 


3.5  The  Kernel  Address  Space 

Each  processor  can  execute  in  oitlior  of  two  address 
5/>aces.  0»»e  is  the  User  >«ddross  .Seace  winch  w.is 
ilescribod  above.  The  second  is  the  hof’u'l  Addre.ss  Space, 
which  is  Similar  to  a user  oddress  space  with  thr»  adcution  of 
some  mechanisms  reserved  for  the  cpo'atmg  sy.stpm  The 
currently  executing  address  space  is  selected  by  a b't  in 
the  Processor  Status  Word  of  the  iSI-ll  A xer^e/ 
Environment  is  similar  to  a Oser  Envurnmonf.  ho.vever 
segments  at  ttie  third  level  of  the  Cupahtlity  List  structure 
(Figure  3 2)  can  bo  User  Pnmary  Capability  Lists  That  »s.  a 
Kernel  Capability  list  structure  can  have  user  environments 
as  substructures. 


Ill  pnrje  15  of  the  kernel  address  space  One  of  these,  the 
L/ser  Environment  regiiter,  holds  an  index  tor  a Capability  m 
the  kernel  envuonment  wlucli  points  to  a user  environment 
This  register  specific:*  the  current  user  environment  for  tins 
processor.  If  the  kernel  writes  a new  index  into  the 
register  the  addressing  state  of  the  old  user  process  is 
saved  by  tl»e  Kmap  in  the  state  vector  part  of  the  old  user 
environment.  The  addressing  state  of  the  new  user  is  then 
loaded  from  the  specifiod  new  user  environment.  The 
addressing  state  is  tiie  value  of  the  window  and  other 
system  registcs  in  pogc  15  of  the  executing  program 
Ideally,  this  oper.iiion.  which  performs  a context  swap  by 
saving  one  adchoss;n(j  state  and  loading  another,  would  also 
save  the  inter'  al  processor  registers.  Unfortunately  tl^ero 
IS  no  woy  for  the  Kmap  to  access  the  internal  registeru  of 
an  LSi-11  Thus  internal  registers  must  be  saved  and 
restored  under  program  control. 


4.  Control  Operations 

The  philosophy  m Cm*  is  to  implement  all  special  control 
operations,  such  as  interprocessor  mterrupts,  by  references 
to  tlie  physical  address  space.  This  not  only  avoids  a 
proLferntion  of  special  control  signals,  but  also  ailov/s  the 
power  cf  the  system's  address  mapping  and  protection 
mechanisms  to  be  applied  to  control  operations. 

The  Slocal  provides  a tliree  pnomy  level  interrupt 
scheme.  An  interrupt  is  invoked  by  wnlmg  into  the 
opprupnate  physical  address  on  the  LSI-11  bus  of  the 
target  processor.  Tiuis  an  interrupt  can  bo  requested  by  n 
process  anywhere  <n  tl  e nefwcrK.  proviued  tJie  process  has 
a Capability  for  a so'jmont  which  maps  to  the  com-'  t 
physical  arldross.  Another  example  »s  the  abort  operation 
If  the  opproprmto  bit  is  written,  a NXM  (Won  Cx'Stont 
Momoiy)  trap  l)V  t!  •'  lo~al  proccs'-or  is  forcoil  Tur, 
mechanism  wtil  ho  used  when  an  error  occurs  durimj  a 
'cmote  roferenco  by  the  processor 

The  follov.nng  exaa'pies  show  how  references  to  sfirNcia 
typed  segments.  or  special  operations  on  standard 
segments,  are  used  to  invoke  microcoded  operations  m the 
Kmap 


4.1  Primitive  Lock  Cpcrotions 

For  procossorn  i.)  the  PDP-11  fomMy.  most  wnli- 
operations  arc  purt  of  a rcad-f' oihf  y- write  seguence  in 
standard  PDP-11s  (..icLfdmg  LSi-1Vs)  this  segucnce  is 
iinplC'Pif»ntr  j as  an  «miivis  oio.  snujlo  bus  operatitm  Tins 
improves  performance  by  reducing  bus  overhead  and 
allowing  op!-.m^  .ilicn  of  references  to  mcmcry  w’f' 
destructive  read  operit  ens  (e  g core  and  dynamic  M05 
memory)  In  C mmp  tr.c  .nd. visibility  of  these  operations  'S 


There  ore  several  additional  pseudo  registers  provided 
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mamtainod  through  the  switcti  to  shared  memory  This  allows 
the  nnplomentation  ot  Locks  and  Semaphores  because  a 
memory  location  can  be  both  tested  and  set  without  tear  of 
an  lotervoniiKj  access  by  some  other  processor.  hu/fv<sJb/o 
read-mociify-writo  operations  to  nonlocal  memory  will  not  be 
iinplomerited  in  Cm*  because  of  ii>creased  bus  and  memory 
contention  and  hardware  complexity  VVe  will  provide  an 
echnvdient  timction  by  making  use  of  the  Kniap's  ability  to 
lock  a segment  descriptor  while  it  makes  a senes  of 
references  to  the  segment.  To  implement  a basic  lock 
mechanism  two  special  segment  oporatio."»s  are  defined 


Inspect  the  v/ord  addressed  If  greater  than  zero, 
then  decrement  Return  the  original  value. 


Incromoni  the  word  addressed.  Return  the  ong.nai 
value. 


4.2  An  fnter-Prccess  Message  System 

Message  systems  can  proviric  particuiady  clean 
mechanisms  tor  communication  between  processes  [B'luch- 
Hansen,  73.  Jefterson  , 77]  In  the  p.sst.  a '^'-4wnncK  xv 
message  systems  has  been  the  sut>«.antlal  oporfitmg 
system  ovorhoad  m transferring  a nu'ssoge  from  one 
process  to  another  .n  a fully  protected  way  The 
architecture  of  Cm*  provides  an  opportun<ly  to  bu'kl  a fully 
protected  message  system  vvhicli  can  be  used  wdn  very 
low  overhead 

A message  port,  or  mail  box  will  be  a spoc'ii  seg*>'ent 
type  Messages  wii  edher  be  entire  segmciUs.  passed  by 
trnnsf  erring  capsbintios.  or  will  DO  single  data  words 
encoded  os  data  capabilities  Two  representollvo 
operations  on  Mailbox  segments  aro 

Send  (Message.  RcpiyMmlBox.  Ma  .Box) 

This  transfers  capabilities  for  a rressage  ond  a reply 
maH  box  from  the  cailer's  Capabil'ty  List  to  the  Mail 
box  If  the  mailbox  is  full  then  the  caller  is 
suspemled 

Receive  (MailPox) 

If  the  mailbox  contains  a riossago  then  a Copabuty 
for  the  message  ano  a Reply  Mailbox  will  bo 
transferred  into  the  cai'er’s  Capability  L'St. 
Otlierwise  the  caller  »s  suspended 

Provided  that  the  above  operations  are  successful,  they 
arc  performed  completely  in  Kmap  microcode,  and  messages 
may  be  passed  with  protiably  less  than  tOO  microseconds 
delay  If  the  operation  connot  be  completed  because  the 
Mailbox  15  full  or  empty,  then  the  operating  system  is 
ifwokod  to  suspend  the  requesting  process  The  Kmop  ran 


also  request  the  operatuig  ' vstem  to  wake  up  a suspended 
process  when  the  operat'  is  complete. 


6.  Development  Aids 

The  development  of  hardware  and  software  for  a new 
computer  system  is  a major  undertaking.  We  have 
attcrppted  to  ease  Ih  s burden  by  using  a variety  of  aids 
All  tl)0  major  hardware  components  were  drafted  using  an 
interactive  drawing  package  (a  version  of  the  Stanford 
OrawiK}  Package).  To  facilitate  the  development  of 
softv/are.  prior  to  the  availability  of  hardware,  a functional 
simulation  of  Cm"  was  programmed,  wliich  executes  on 
C mmp  Ceyeiopmc’nl  of  the  Kmop  hardware  and  microcode 
has  been  greatly  benefited  by  the  use  of  tfie  "hooks" 
mochan.sm  'n  the  Kmjp  This  connection  to  the  Kmap 
allows  a program  execut  ig  on  an  LSI-1  t almost  complete 
access  to  the  internal  state  of  the  Kmap. 

In  order  to  expcd  tc  htt'dware  debugging  and  software 
dpveopmrnt.  a host  program  dcve'opmcnt  system  wns 
constructo'i  The  nost  is  connected  to  each  Cm  n the 
‘vstem  H Scr  a*  Lme  U ><t  (SLU)  to  ailo-w  down  line  memory 
loaoi'M  ''’d  djTong  from  tho  associated  Cm.  In  arlciitlon 
ti^e  S>U  moKOS  consO'C  control  functions  for  each  LSI- 11 
ava  able  t)  tne  host  computer  [van  Zoren.  75]  The  Host  m 
turn  IS  Connected  to  a POP- 10  timesnar.ng  system. 


6.  Concludmj  Reir.arks  and  Project  Status 

Cm'  IS  projected  to  oc  constructed  m three  stages  The 
UfiA  Stage  is  a ten-processor,  three  Kniap  system.  The 
subsequent  stages  v^.-l  include  30-procossors  and  later 
1 0O-pfoccssom.  Detailed  hardware  design  begun  in  late 
July.  1975  As  of  intc  summer,  1976,  a throe-processor 
or^e-Kmap  system  was  operational  It  is  expected  ti>ot  tl'L 
first  stage  Cm'  configuration  will  be  operational  m the 
second  (junrtcr  of  1977  The  initial  operating  system  m 
dc5cri‘)cd  in  [Jones,  et  ai  77]  and  is  being  developed  »>»'*•’ 
on  the  Cm'  simulator  which  runs  on  C mmp  and  on  tlu»  tt-  e 
hardware  with  the  support  of  the  Host  Dovciopmcn*  sysi‘*"- 

The  essential  teattircs  of  the  Cm"  architocture  ha.i 
becr>  presented  Both  the  coupl.ng  of  a processor  d<re'.t  ^ 
witli  c'ach  unit  of  sluircd  memory  and  the  three  level  h 
structure  wl>ich  makes  all  m<>mory  accessible  bv  o.'e». 
processor  are  priiiuiry  'eatoros  of  the  Cm'  structure  M i;  n 
of  the  soplusticQt.on  ,n  tiie  architecture  is  associa'ed  w in 
the  address  translation  mechanisms  A dc  scnption  has  bot-n. 
givon  of  hov/  the  smo"  processor  address  space  c*  mo 
POP- ? 1 IS  ma.opcd  into  the  larger  yiohal  virtual  ndrlrr**,s 
si>acG  of  the  Cm'  system  and  how  the  g'obnl  virtual  address 
space  IS  mjppcd  onto  the  distno.jlod  physical  address 
space  of  the  Cm'  system 
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A number  of  important  aspects  of  the  Cm"  project  ore 
outside  U»c  scope  of  this  paper  and  interested  readers  are 
referred  to  other  papers  for  a more  complete  discussion 
[Jones,  et  al.  77,  Swan,  et  a!-  7GA,  7GB,  77.  Ingle  and 
SicwioroK,  76A,  Ingle  and  Siewiorck,  7GB,  Siowiorek.  et  al 
76],  Reliability  and  performance  models  have  been 
developed  concurrently  with  the  hardware  design  of  the 
system  and  have  been  used  to  guide  several  important 
decisions  concerning  the  structure  of  the  Cm* 
implementation. 
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Abstract 

Tile  implomentatton  of  a hierarchical,  packet  switched 
mulliprocossor  is  presented.  The  lowest  love)  of  the 
structure,  a Con}putQr  Module,  ts  a processor  ^memory  pair. 
Computer  Modules  are  grouped  to  form  a duster; 
communication  within  the  cluster  is  vid  a parallel  bus 
controlled  by  a centrnlired  address  mapping  processor. 
Clusters  comniunicato  via  iiitercluster  busses.  A memo''y 
reference  by  a program  may  be  routed,  transparontly,  to 
liny  memory  in  the  system.  Tills  paper  discusses  tlie 
hardware  used  to  iniploment  the  comrmmication  mechanism. 
The  use  of  special  diagnostic  hardware  and  performance 
modda  is  also  discussed. 
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1 Introduction 

The  companion  paper.  [Swan  et  al  1977],  has 
introduced  Cm"  as  a large,  extensible  multiprocessor 
architecture  It  has  an  unusually  powerful  and  complex 
addressing  structure  winch  allows  close,  protected 
cooperation  between  large  numbers  of  Inexpensive 
processors.  This  paper  describes  the  combination  of 
hardware  and  firmware  which  implements  the  address  space 
sharing  and  intcrprocessor  communication  mechanisms 

Cm*  Is  a multiprocessor  system  f\s  we  define  it  (rather 
than  a network  of  independent  computers)  because  Uie 
processors  share  a common  address  space  All  processors 
have  immediate  access  to  all  memory.  The  structure  of  Cm" 
Is  shown  In  Figure  11.  The  primary  unit  Is  the  Computer 
Module  or  Cm.  This  consists  of  a processor,  memory  and 
peripherals  interfaced  to  a local  momory  bus  and  a "local 
switch"  The  local  switcli,  or  Stocat^ , interconnects  the 
processor,  its  local  memory  bus  and  the  Map  Bus.  The  Map 
Bus  provides  communicotion  between  up  to  fourteen 
Computer  Modules  witiun  a duster,  and  is  centrally 
controlled  by  the  Kmap,  a high  performance 
microprogrnrn.Tied  processor.  Each  Kmap  Interfaces  to  two 
Intcrcluster  busses,  by  means  of  which  it  communicates  with 
the  other  clusters  >n  the  system. 

There  is  a system-wide  28  bit  virtual  address  space 
Tins  address  space  is  divided  into  segments  with  a max/mi/m 
Size  of  4090  bytes.  Programs  refer  to  segments  indirectly 
Via  Ciipab///tres.  winch  ore  Iwo-word  items  containing  the 
giobnl  name  of  a segment  and  specifying  access  ng’its  to 
the  sofjmrnt  The  processors  have  a 1G  bd  address  space 
which  IS  divided  into  10  pages  A mechanism  Is  provided 
which  allows  a program  to  associate  any  Capability  it 
posesses  (and  hence  any  segmt'ot  to  which  it  is  allowed 
occGss)  with  any  page  m its  immed  ote  address  space  A 
full  description  of  the  address  mapping  scheme  is  given  in 
[Swan  et  nl  1 977] 

To  demonstrate  the  viobuity  of  a structure  it  is 
necessary  to  tiuild  a pilct  system  with  currently  available 
components.  To  bo  a successful  demonstration,  the  pilot 
System  has  to  bo  o useful,  economical  computing  resource 
111  its  own  right.  Therefore,  in  the  Cm"  network  descnlicd 
here,  many  design  tradeoffs  were  made  on  tho  basis  of 
currtMit  lecfuiolo'jv  mui  the  resources  availahio  The  liighly 
expcri-nentol  nature  of  tho  project  encouraged  an  Grrphasis 

^The  names  used  for  hardware  components  of  Cm*  are 
derived  from  PMS  notation  [3cil  and  Nowell.  71]  The 
Iradirv),  cnpitali.’<?d  letter  mdiratos  the  primory  function  of 
the  unit,  eg  Computer.  Procr'ssor.  ko^trollo^  Link.  Switch 
The  suhsegvient  lette's.  optionally  sr’pnrnted  with  a perioij. 
give  .some  attnlxite  of  the  unit  For  example,  Slocnl  is  a 
'ocnl  switch  Pninn  s n mopping  processor  The  name  rm" 
dorivrs  from  (Ccmpide*  •nodular)*  where  * is  the  Moer>o 
stor 


PciOe  2 


The  Implementation  of  the  Cm“  Multi-Microprocessor 


I 

i 

j 

1 

I 

I 

j 

i 

■i 

r 

r 


;( 

i, 


Intercluster  Bus 


I 


'I 


Figure  1.1  A Simple  3 Cluster  Cm*  System 

on  generality  and  ease  of  debunrvng  in  tlie  hardware 
components,  rather  than  just  inimmiirution  of  costs.  Tiiere 
ore  mony  aspects  of  the  detailed  dostgn  which  would  iinve 
to  be  ro-evaluated  if  tlie  structure  were  to  be  imptemented 
in  a different  technology  or  built  os  a commercial  product. 
In  particular  the  distribution  of  functions  between  the 
proressois  and  the  Kmup  would  bo  cnrefully  reconsidered. 
The  modular  nature  of  Cm*  ii’akes  it  particularly  suitable  for 
implementation  in  LSI. 

Scctfon  2 lUustrafes  t)ie  mochanisn)  for  memory 
references.  Tho  various  hardv/are  components  of  Cm*  are 
described  in  the  following  six  sections.  Section  3 descrii>es 
tlie  processor-memory  pairs  and  thG>»'  interface  to  tho  Map 
Rus  In  Section  4 opportunities  for  pnrallellsm  in  the 
address  mapping  mechanism  are  considered  Three 
autonomous  functional  units  of  the  Kmap  are  presented  in 
Sections  5.  6.  and  7 Section  3 describes  the  support 
(jivt>n  to  hardware  diagnosis  and  miciocodo  development  *;> 
tho  Kmap  For  an  effective  implementation  *t  was 
necessory  to  find  a reasonable  porrormance  hnianro 
bf?twron  system  components  Some  of  the  pe'^orrnrico 
rnoarHIing  wt>ich  guided  our  judgomunt  is  presented  in 
Suction  9 


2 Tfte  Mechanism  for  Local  and  Nonlocal  References 

Addresses  cjoncratod  t'y  i.rocnssors  in  n Cm*  system 
nniy  refer  to  fr:<*nory  oi^ywhere  wt'iin  the  system  Manpinj 
of  an  address  and  routing  to  the  appropriate  remory  Ofu 
pcrformc?d  m a way  that  is  totally  transparent  to  t)u» 
processor  generating  tt  e audress  If  an  address  is  to  refer 
to  the  memory  local  to  that  procossor.  tho  momory 
reference  is  portorru*d  ir.  a completely  standard  wav 
except  tlwil  the  Slocal  reoentes  the  h.gn-order  four  bits  of 
the  address  See  Figure  ? \ 

When  tiu'  page  hein  r rnreronred  m not  irrcnl  ('»'  Ibn 
“Map  ' t)'t  for  the  re'nrencuil  pagn  s set  in  the  SlOCnO  n 
se^u/ce  ruf/t/esf  is  mad'  «c  the  Kmnp  oy  the  S'ccnl  LJpon 
'eccivmg  the  servu  e re  juest  the  kmap  executes  a Miw* 
bus  r.vcle  to  read  .n  t'n'  processor*n'*nernt0v1  address  f»(  r 
the  5locnl,  as  weM  as  the  nun^bo*  o'  the  Cm  maVmg  tnn 
requost.  and  tv.'O  stnbis  l ds  indicating  wuth  address  sp.i*,e 
was  executing  on  t'u  I'oressor  and  whether  th«»  rprerenro 
was  a read  or  <,  w'  te  is«*M  Fig. ire  2 2)  Jf  the  segment 
he  ng  referf*nce.»  is  a«  to  the  Cluster,  the  kn>ap  wiH  use 
idformat'cn  cached  n -ts  nigh-sr'cod  Duf'ors  to  bypass  reost 
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Figure  2.1 

Addressing  Mechanism  for  Local  Memory  References 

of  the  processor-to-virtual-to-physical  address  mapping. 
Thus  It  can  quickly  translate  from  the  page  number 
referenced  by  tho  processor  to  a physical  address 
consisting  of  the  number  of  the  Cm  containing  the  physical 
location  and  an  eiqhteen*b:t  local  address  A second  Mop 
Bus  transaction  is  executed  to  i>oss  this  address,  and  a hit 
indicating  whether  a road  or  o write  is  to  be  performed,  to 
the  destination  Slocal  If  the  operation  is  a write,  the  data 
may  bo  passed  directly  from  tho  Cm  making  the  reference 
to  tho  Cm  containing  the  word  to  be  written  The 
destination  SIocqI  performs  the  rend  or  write  via  a Direct 
Memory  Access  When  this  is  completed  it  issues  a return 
request  to  the  Kmap  to  acknowledge  completion  A third 
Map  Bus  cycle  is  performed  to  transfer  Iho  dato  back  to  the 
processor  that  made  the  reference  (in  the  cose  of  a read) 
and  to  acknowledge  completion  of  the  reference  so  thot  the 
requesting  processor  may  resume  activity 

A second  alternative  wlicn  tho  Kmap  receives  an 
address  to  map  is  that  the  physical  location  being 
referenced  Is  not  local  to  the  duster  In  this  case  the 
information  cached  in  the  Kmap  for  the  page  being 
referenced  will  not  indicate  a physical  location  directly; 
instead  it  will  give  a sixteen-bit  segment  name,  tho  number 
of  the  cluster  containing  the  physical  memory  allocated  to 
the  segment,  and  two  bits  used  to  extend  tho  read/wnto 
bit  to  0 three-bit  op  code.  This  information  is  combined  with 
the  twelve  tov^-order  bits  of  the  oncjiniil  processcr  address 
to  form  the  full  virtual  address  of  the  object  being 
referenced  Seo  Figure  2 3.  The  virtual  address,  along  with 
the  processor  data  (if  a write  is  being  performed)  is  sent 


Interclustcr 


Source  Cm  Destination  Cm 

Figure  2.2  The  Mechanism  for  Cluster-local  References 

via  an  Intercluster  Bus  to  tlie  Kmap  of  ttie  duster  contuining 
the  segment  (if  there  is  no  Intercluster  Bus  directly 
connecting  the  fivo  Kmaps  the  mossago  will  be  steeroO  from 
Kmap  to  Kmap  liiUi)  it  reaches  the  dostinotion  cluster).  The 
destination  Kmap  wiM  then  map  tho  virtual  address  to  a 
physical  one  within  its  duster.  Map  Dus  transactions  will  be 
executed  to  pass  t/io  phy.sical  address  (and  data  if 
needed)  to  an  Slocal  which  in  turn  performs  the  operation 
and  returns  acknowicdgcmont  (and,  perhaps,  data)  back  to 
tlie  destination  Kmap  A return  message  Is  used  to  pass 
back  acknowicdgcmont  and  data  to  the  Kmop  of  the 
orlqii>nting  duster  Finally,  this  Kmap  vviK  relay  the  data  and 
acknowledgement  back  to  the  initiating  Cm  to  complete  the? 
reference. 

SovornI  point.s  are  worth  noting  with  respect  to  Tho 
above  schemer,  Cxccpt  at  the  local  memory  bus  level, 
whore  conventional  circuit  switching  is  used.  ah 
conmuimcation  Is  pcrformocl  by  packet  switching  That  is. 
busses  nro  nllocntod  only  for  tlir  period  reqim(»d  to  trnnsf»*r 
data.  The  data  is  latched  at  each  interface,  rather  than 
estabhslnng  a continuous  circuit  from  tho  source  to  the 
destination.  This  approach  gives  greater  bus  utill^otion  and 
avoids  cieediod.  ovi''’  bus  allocation  All  transactions  are 
completely  interlockoil  with  positive  acknowledgement  being 
re(|iurei  to  signal  con-piction  of  an  operation  (it  is  possible 
to  allow  a processcr  executing  a nonlocal  wnte  to  procei'ti 
as  soon  ns  tiui  clnta  for  the  wr*le  l»as  been  received  by  the 
Kmop  or  dostinotion  .Slocal.  wdliout  wading  for  completion  of 
flip  opcrntion,  however  In  this  case  tJte  Kmop  wui  expect 
to  receive  acKnov/irdqenu’nt  in  place  of  the  processor  so 
tliat  appropriate  actions  may  bo  taken  if  none  is  received) 
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The  complete  proc  cs  sor  ~ to- vjrtual- to- physical  address 
mappinn  per'ormed  only  in  the  case  of  intercHistor 
references.  As  the  locality  of  a reference  Increases  the 
amount  of  tins  mapping  that  may  be  bypassed  (and  hence 
the  speed  of  khe  reference)  increases,  with  local  caches  of 
certom  mapping  Information  used  to  effect  the  bypass.  An 
important  charactonst*c  of  the  addressing  structure  is  that 
there  is  exactly  one  Kmap  that  moy  perform  the  virtual-to- 
physical  mapping  for  a given  segment  The  requirement  that 
all  references  to  a segment  occur  with  the  cognizonco  of  a 
single  Kmap  greatly  simplifies  the  moving  of  segments  and 
the  implementa  tion  of  operations  roqufring  mutual  exclusion, 


3 The  Computer  Module 

The  first  level  of  the  Cm*  network  hierarchy  is  the 
Computer  MocIjIq.  or  Cm.  The  Cm’s  provide  both  the  memory 
and  processing  power  for  the  multiprocessor  system 

The  decision  to  use  a standard,  commercially  avmlahlo 
processor  (the  DEC  LSl-tl)  has  had  a considerable  impact 
on  the  design  Use  of  a standanJ  instruction  set  has  made  a 
hirgc  poo!  of  software  and  software  development  aids 
directly  available  The  not  inconsiderable  effort  to  design 
and  implement  o new  processor  has  been  avo'dod 


At  the  software  level,  the  prime  disadvantage  of  the 
LSl-1  1 instruction  set  is  that  only  16  bit  addresses  can  be 
directly  mon'piilated  The  companion  architecture  paper 
discusses  in  detail  the  mechanism  used  to  expand  a 
processor's  address  space  from  16  bits  to  28  bits 


3.1  The  Components  of  a Computer  Module 

A Computer  Module.  Figure  3 1,  can  act  as  a stood  alont' 
computer  system.  The  standard  commercially  availohlo 
components  include  the  DEC  LSl-11  processor  and  dynamic 
MOS  memory  Ariy  LSI*  11  peripheral  may  be  used  on  the 
bus,  incKid.ng  setiO'  and  parallel  interfaces,  floppy  and  fixr*:l 
head  rl'sks.  etc  Tlie  standard  10  bit  memory  has  boon 
Gxtenried  with  byte  parity  Memory  refresh  is  normaMy 
performed  hy  microcode  m the  LSI-11,  however,  the  fact 
that  a f>rr>ces.sor  r-)oy  be  su'.prnded  indefinitely  wht’o 
owaiting  tiio  ccmplelipn  of  u complex  external  '^efc>r(*nc  e 
has  made  d necessary  to  augment  each  Cm  with  a special 
bus  dev  ee  to  perform  -^efresh 

The  mest  importni't  component  wluch  has  been  added  to 
each  Cm  »s  the  Siocai  T 'is  provides  the  mterfoce  between 
the  processor,  the  V.id  Hvis  and  thn  vSl-11  Bus.  The  pf»'*’i' 
functfon  of  frv  SlOC.iJ  <s  to  seioctivi  y pass  re*«»rencMS  frr^n' 
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Figure  3.1  Details  of  a Computer  Module 

tlie  processor  to  cither  the  LSI-11  Bus  or  the  Mop  Dus  and 
to  accept  references  from  the  Map  Bus  to  the  LSi-11  Ous. 
The  Slocal  also  provides  simple  address  rolocotion  for 
references  made  by  its  processor  to  local  memory.  Figure 
2 1 shows  l)ow  this  relocotion  Is  performed;  the  "Map  Bit'*  In 
the  local  relocation  table  «s  set  for  pages  which  are  not  In 
the  local  momory  of  the  processor 

In  addition  to  the  Local  Relocation  Tehle  the  Slocal 
provides  n miinber  of  other  control  registers.  All  these 
registers  nro  ndrlrossahle  os  memory  locations  on  the  LSI- 
1 1 t)iis,  however  only  the  Kmop  and  highly  privileged  system 
code  wiH  have  direct  access  to  them  One  of  the  key 
registers  is  the  cVternn/  Processor  Sfdfi/s  Word 
(Xf^SW<l5  8>)  The  LSI- 11  imp'Cments  only  the  lov/  order 
byte  of  the  standard  PDP-11  Processor  Status  Word 
(PSW<70>).  Logic  in  the  Slocal  (with  assistance  of 
standard  signals  from  the  LSI*11)  allows  the  XPSW  to  be 
saved  and  restored  during  interrupt,  trap  and  other 
operntions  in  unison  with  the  internal  PSW  The  XPSW 
allows  selective  cnabiiiuj  of  vnrious  Slocal  functions  and 
controls  n simptr*  three  level  interrupt  scheme  On  power-up 
the  XPSW  »3  cleared,  which  disables  all  special  operations 
l)y  the  Slocol  including  the  ro'ocation  of  local  memory 
references  In  this  mode  tiu*  processor  acts  os  a l)n^e. 
unmodified  I SI- 11  The  Local  Relocation  Table  con  be 
initinli '•ed  either  by  console  operations,  execution  of  local 
bootstrap  code  or  remotoly  by  any  processor  in  the 
network.  After  initMli/ation.  enabling  Rnloc  Mode 
(XPSW<1  1>)  will  allow  local  relocation  and  give  access  to 
the  rest  of  the  network. 

Incorrect  u.se  of  PDP-11  instructons  such  as  HAt.T. 
RESt  T,  Move-To-Piocess  jr-Stntus  - word.  Return  from 

Interrupt,  etc  can  causo  loss  of  a processor,  garbling  of  an 


I/O  operation  or  enable  circumvention  of  the  system's 
protection  scheme  The  Privileged  Instruction  Mode  bit 
(XPSW<13>)  enahlGO  logic  in  the  Slocal  which  detects  the 
fetching  of  any  "dangerous"  instruction.  An  immediate  error 
trap  IS  forced  if  an  unprivileged  program  attempts  to 
execute  o privileged  instruction. 

Several  registers  m the  Slocal  are  concerned  with 
providing  diagnosis  and  recovery  information  after  a 
software  or  hardware  error  is  detected.  Almost  all  errors 
ore  reported  to  the  processor  by  forcing  a NXM  (Non 
eXistent  Memory)  trap.  This  includes  errors  detected  by  the 
Kmop  during  remote  references.  The  Kmop  signals  the  error 
by  writing  to  the  "Force  NXM"  bit  In  an  addressable  register 
In  the  Slocal  The  Local  Error  Register  indicates  the  nature 
of  the  error  and  whether  the  erroneous  reference  was 
niopprri  Tiie  "Last  Fetch  Address"  register  is  updated  to 
hold  the  address  of  the  first  word  of  an  instruction  every 
time  the  LSI-11  fetches  a new  Instruction  If  an  error  is 
detected,  this  register  is  frozen  until  the  Local  Error 
Register  is  explicitly  cleared  Also  frozen  in  the  Local  Error 
Register  is  a count  of  the  number  of  memory  reforonces 
performed  in  the  execution  of  the  instruction.  In 
conjunction,  these  two  registers  provide  sufficient 
nifonnal»on  to  restore  the  state  of  the  LSI- 11  tor  retry  of 
the  instruction  during  which  the  error  was  detected 

Tlie  Slocnl  also  provides  two  interrupt  request  registers 
Interrupt  enable  bits  m the  external  processor  status  word 
allow  masking  of  the  interrupt  requests  Provided  roforenco 
is  permitted  by  the  memory  protection  scheme,  any 
processor  in  the  network  con  interrupt  any  other  processor 
simply  by  writing  to  the  correct  odiJress 


3.2  Data  Paths  for  Nonlocal  References 

An  Klcsl'zcd  form  of  tne  basic  data  paths  and  latches 
within  a Cm*  cluster  shown  in  Figure  3.2  Depending  on 
the  address  generated,  a raference  from  the  processor  is 
passed  edher  to  the  local  memory  {)us  or  to  tlie  Mop  Bus  A 
local  memory  reference  »s  performed  in  a conventionol  way 
For  A nonlocal  reference,  the  address  (and  possibly  dota)  is 
latched  and  a service  rcq.icst  is  issued  to  the  Kmop 
broken  line  m Figure  3 2 shows  the  path  of  a read  to  the 
memory  of  onother  Cm  M tl.o  cluster  The  address  from  tiui 
source  processor  is  ruad  by  the  Xmap  wh'ch  translates  *t 
into  a physical  oudr.'ss  wdlun  the  memory  of  a Computer 
Modulo  This  physical  addn'ss  is  placed  onto  the  Man  Bus 
by  the  Kmap  and  'atched  at  ^he  target  Cm  A converitional 
D»rr?ct  Motiory  Access  (DMA)  cycle  'S  performed  by  the 
destination  Slocal.  the  data  read  is  hitched  ond  tiu*  xnmp  i% 
ngam  requested,  tlus  t*M*e  with  a return  request  To 
comp'ote  the  operation,  the  Kmap  responds  by  transfern  »q 
the  data  ever  the  r/ar>  Mis  tr^n  tr>*»  target  Cni  to  the 
requesting  Cni  illus  smiply  requires  the  latch  nt  tlu^  Tarrjrt 
Cm  to  be  enabled  onto  ttu»  Mai>  Dus  and  the  latch  at  the 
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Figure  3.2  An  Idealized  and  Simplified  Representation  of  the  Data  Paths  in  a Cluster 


Map  Bus 


Figure  3.3  Simplified  LSI-1 1 - Slocal  Data  Paths 
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rt?<|uostmg  Cm  to  bo  strobocO  At  t»ns  point  tin*  sourco 
processor,  wvhicb  was  sospencieci.  is  (jwen  tlie  data  /15  if  ci 
nornuf  nu>rnor/  (cforonco  haO  boon  por'ormod 

Thn>  simphfiocJ  (Joscni)tion  of  a Compiitor  Module  has 
born  presented  to  cnphasizo  tl»e  sm’plicity  of  tlie  basic 
mocitanisms  required  tor  an  intro-clustof  rofnronco  in  Cm*. 
In  the  actual  miplcmentation  using  the  L6I-1  t processor  the 
<l(ita  paths  ore  rather  diftoront  than  the  idouluod  structure 
stiown  m Figure  3.2  The  differences  are  due  primarily  to  ttie 
need  to  minimize  the  changes  to  the  LSI- 11  Although  stiM 
Simplified,  Figure  3 3 1$  a more  accurate  representation  of 
tlu»  data  patits  and  latches  used  to  interface  the  LSI- 11 
and  the  LSI-1  1 bus  to  tho  Map  Bus 

I he  processcjr  board  is  modified  so  that  the  local 
Hfijcation  Table  in  the  SlOCfll  can  ue  inserted  m the  data 
PtJth  of  the  four  high  order  address  bits  Tito  tn'm'g  :na'*.;  ns 
in  the  i>roci'ssor'3  address  path  are  wide  enough  to  at'o/y 
insertion  of  tins  do’ay  without  loss  of  [>or*orniance.  Tiu? 
LSI-1  1 Bus  IS  tluj  only  data  path  from  the  procossrr  for  both 
local  and  non  local  references  tf  the  processor  were 
permitted  to  hold  the  LSl-11  bus  vvlu'c  waituig  for 
completion  of  a noitlocal  reference  then  references  from 
other  processors  in  the  nelworK  to  momo'’v  on  l!»e  LSl-11 
l)us  would  be  hiocLotl,  This  could  very  easily  lead  to 
deadlock  situations.  To  give  greater  concurrency  and  to 
eliininato  tite  doadiock  potential,  tlio  Siocnl  is  able  (usmg 
smii'to  microcodod  state  seciuence  logic)  to  force  the 
processor  off  tlie  I SI- 11  bus  w-ulc  It  is  waiting  for 
completion  of  nonlocal  references  While  the  proct'ssor  is 
forced  off  the  local  bus  tl’.o  Siocai  ta^es  over  DMA  bus 
arbitration  for  the  susponood  processor 


4 Concurrency  witlun  the  Mapipiruj  Mechanism 

tnrly  in  the  design  of  Cm*  the  soccds  of  tho  various 
comroiu'nts  m the  system  honan  to  ippr  ir  as  fcdicws  th.o 
tiiiio  for  a ‘'typical”  Map  fius  trao',>a<.t.OM  was  about  0 
niicros«*condS;  tho  tnno  required  in  tlic  computational  unit  of 
the  Kmap  for  an  address  mapping  was  1-2  mic  rOsSeconrts. 
the  tinu'  to  transfer  a oiossncie  on  on  intercl.ister  Bus  was 
2-4  niicroseconUSi  and  the  tune  for  an  Siccol  to  execute  a 
read  or  write  requested  by  the  Krrmp  was  3-4 

n’ir'’or»<*coints.  In  rofernm)  to  the  mpcnanisms  for  nonlocal 
nMppinqs  it  can  bc  seen  thut  no  S’nqlo  component  is 
r'^nncnMhic  for  a very  large  fraction  of  the  time  rogu-rrMl  for 
a nonlocal  rcforr'nce  Thus  if  each  cluster  had  a mapp.ng 
concurrency  of  cat*  (only  one  nonlocal  ro^oronce  could  be 
proci'ssed  at  a time  per  cluster)  tioth  tl'.o  utiluation  of  the 
components  and  the  tnrmirihput  of  the  meclianism 
wouhj  be  low  (the  ef'oct  of  concurrency  on  system 
l>cr'ofr'jtice  IS  discussed  quantitatively  rn  Section  In 
adri.tion  tk.e  por.sih.idy  of  dondiDCk  m intcrc'uster 
refer«Micer»  in  introduceil 
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exclusion  (for  example,  changing  the  virlual-to-physical 
mapping  of  the  system)  will  be  impicmentccJ  in  Cm*  as 
memory  references  to  "special"  segments  which  will  then 
cause  the  Kmap  to  perform  the  ocsired  operations  in  a 
protected  way  In  generaJ  these  operations  will  require 
several  references  by  the  Kmap  to  mam  memory  If  the 
Pmap  is  to  be  used  for  other  mappings  while  these  rraln- 
memory  references  arc  being  mode  by  the  Kbus  and  Slocals, 
there  must  be  some  means  of  saving  and  restoring  Its  state 
so  that  processing  can  bo  resumed  when  the  memory 
reference  has  been  comploteo.  The  solution  adopted  Is  to 
provide  registers  in  the  Kmop  to  save  and  restore  state  for 
up  to  eight  overlapping  operations  A mapped  operation  in 
some  stage  of  processing  by  the  Kmap  is  referred  to  as  a 
confexf.  Each  context  has  ailocotod  to  its  exclusive  use 
eiglU  gcnornl-purpose  registers  and  four  subroutine  linkage 
registers  (one  of  which  Is  used  to  save  the  microprogram 
address  while  awaiting  the  completion  of  Map  Bus 
transactions). 

The  Kbus  maintains  the  statute  of  the  eight  Pmap 
contexts  and  allocates  them  to  new  service  requests.  The 
context  number  and  other  status  arn  then  placed  In  the  Run 
Queue  to  signol  the  Pmap  that  tlie  context  is  runnaolo  The 
mapping  processor  «TCtivates  the  context  by  removing  Its 
number  from  the  Run  Queue  and  starting  execution  of 
microcode  at  an  address  determined  by  the  status  bits. 
When  the  now  context  is  activated  the  processor  address 
Is  mapped,  and  a request  for  a n’am-memory  refcrenco  Is 
placed  in  the  Out  Queue  (during  this  tin*e  the  Kbus  has  been 
tree  to  read  in  service  requests  or  perform  functions 
recpicstod  by  the  Pmap)  A conft?»f  swap  is  executed  in  the 
Pmap  to  deactivate  the  current  context  pending  the 
completion  of  the  memory  '■eference  and  to  activate  the 
next  one  in  the  Run  Queue.  The  Kbus  transfers  acJdres'i  and 
data  to  the  destination  SlocnI.  then  processes  other 
requests  while  tt»e  memory  reference  is  being  performed. 
When  llio  memory  reference  is  con^plcted  the  Kbus  cither 
rends  the  acknowledgement  and/or  data  back  into  the  Kmap 
and  places  the  context  back  In  tlie  Run  Queue  for 
reactivation,  or  it  sends  the  acknowl.rdgement  back  to  tlie 
processor  that  orignuiMy  inaoe  the  service  request  (thereby 
completing  the  mapping  operation)  and  marks  the  assocoted 
context  as  "free"  for  reallocation  to  a new  service*  request. 
The  fact  that  n context  remains  allocated  to  each  nonlocal 
reference  until  that  reference  -s  completed  (regardless  of 
wheUicr  or  not  more  Pmap  processing  is  expected  to  be 
needed)  means  that  if  an  error  Is  detected  the  context  con 
1)0  reactivated  and  will  liavc  enough  state  information  to 
handle  the  error  in  an  intcllicjcnt  fashion. 

Communication  between  the  Lmc  and  Pmap  is  similar  to 
that  between  the  Kbu.*?  and  Pmap,  the  Pmup  queues  a 
re.iuest  for  an  intercUister  message  to  be  sent  (separate 
queues  ore  provided  for  each  Intercluster  Bus)  and 
suspends  the  requestnuj  context  When  a return  messaqe 
IS  received  for  the  context  the  Line  causes  the  Kbus  to 
reactivate  the  context  in  the  Run  Queue  When  an  incoming 


mtercijster  message  is  received  by  one  of  the  Line's 
Intercluster  Bus  Ports,  it  is  queued  and  a request  is  issued 
to  the  Kbus  to  allocate  a free  context  to  the  request  and 
activate  it  in  the  Run  Queue. 


6 The  Kbus  and  the  Map  Bus 

Because  of  the  groat  variety  of  tosks  it  must  perform 
and  the  necessity  that  it  be  able  to  respond  to  errors  In  an 
Intelligent  way,  the  Kbus  was  designed  as  a 
microprogrammed  processor  controlled  by  256  40-bit  words 
of  read  only  memory  It  has  a microcycle  time  of  100 
nanoseconds  which  is  synchronized  wltli  the  1 50 
nanosecond  clock  of  the  Pmap  and  Line  at  50  nanosecond 
intervals  Figuro  5 1 shows  the  major  dements  of  the  bus 
controller 

The  Mop  Bus  contoins  38  signals,  of  wlhch  20  are 
bidirectional  lines  used  to  transmit  addresses  and  data 
between  the  Slocals  and  Kl)us  of  tlie  cluster  The  Kbus  is 
master  of  all  transactions  on  the  bus.  as  such  it  specifies  n 
source  and  destination  for  each  cycle  os  well  as  status  bits 
indicating  the  use  of  tlie  data  (address,  data,  etc.)  The 
bus  IS  synchronous,  with  the  Kbus  generating  all  of  the 
strobes  used  to  transmit  data  Each  Slocal  is  provided  with 
private  service  and  return  request  lines  to  the  Kbus  The 
arbiter  section  of  the  Kbus  scans  these  In  a pseudo  round 
robin  priority  scheme. 

The  Kbus  maintains  the  (;ueues  and  rogistprs  used  for 
comnuimcation  with  the  Pmap  The  Run  Queue  contains 
eight  ciyht-bit  slots  (and  t.hus  is  g;;arantoed  never  to 
overflow),  eacli  containing  a threc-b  t context  and 

five  nfhJtional  b-ts  o*  activcition  status  The  Out 
contains  four  J9-bit  rntr*rs  The  Pmap  loads  ttils  queue  to 
roquest  Kbus  operations  and  must  check  its  stale  be^o»e 
loading  to  insure  that  it  never  overflows  Each  Out  Queue 
slot  contains  an  op  rode  used  to  select  one  of  thirty-lw"' 
Kt)us  opCMition.s,  nrul  aclclitionnl  address,  dat^  and  context 
information  relevant  to  the  operation  Two  registers  aro 
loaded  by  the  Kbus  on  behalf  of  each  Pmap  context  They 
arc  readable  only  by  the  Pmop  and  writable  only  bv  the 
Kbus  The  Bus  Ptif-n  Regtster  contains  the  last  da*a  w utj 
road  in  from  U.'’'  Map  Bus  for  the  context  and  ttm  fin.s 
Condition  R<^fjistvr  gives  control  and  stotus  information  fc' 
the  transaction. 

Tfio  Kbus  IS  responsil)h'  for  the  aHocotion  and 
deallocation  of  cuntexts.  and  mainioins  the  statiis  -f  eai  ’ 
context  for  tn  s purpose  It  a^so  Keeps  two  ad'Micnnt  hi*, 
of  status  for  each  context  which  are  used  to  insure  tea* 
wiM'n  fl  context  s isju  nds  itself  tn  awa  t Ihn  rxecutior)  ♦ a 
n»ain-momor/  ntference  or  th«  send,  iq  o*  an  intorrhis’er 
mrssane.  nn  ac  Wn.iwieuqemrrit  of  the  c.imi.ipt  a of  th«- 
0|)cratn  n is  re.-.c.^e'J  w thin  a rosse  iab'e  tim^  It.v' 
milliseconds)  f a susurruled  context  t.-u  s out  it  -s  forciu  v 
reactivated  w th  status  1 ts  indicating  the  error 
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Service  and  Return  Queue' from  Line 


Figure  5.1  The  Components  of  the  Kbus 


bes.so.s.  take  d<itrt  from  vonotis  sources  nnd  feed  Iho^  to 
The  Khns  also  mamtairus  n.ne  l^ts  of  status  tor  each  the  inputs  cf  tlie  Ar.tliint  t-c  Lorjic  Uiut  The  third  bus.  the 

Sloc;al  In  file  clustof  maicitmg  wiicflier  file  Slocal  is  busy  f Bus.  lakes  the  All)  output  and  distnljutes  it  to  various 

with  a Kmap-requested  menory  referonce  and,  if  so,  what  parts  of  the  Kniap  The  Kbus  and  Line  are  also  connectod  to 

to  do  with  the  inlorniation  returned  at  the  end  of  the  these  busses.  Pipeline  latches  are  used  to  overlap  fetch  of 

traii.saetion.  This  status  is  set  whenever  a local  niemory  operands  With  currnnt  data  operations 

rL*f(.*ronco  is  initiated  and  is  used  to  msuro  that  two 

contoxts  do  not  simuifaneot/sly  try  to  request  a memory  Tiu?  Sh>n  jr>d  Umt  provides  the  ability  to  por'om 

access  through  the  same  Slocal.  tie'd-oxtraction  on  one  of  the  A1  U operands  This  copnt)itdv 

IS  important  since  tiie  P(»;ap  froquentiy  deals  with  packtxJ 
information  in  scqau’nt  descriptors,  intorchjster  messages, 
etc.  The  input  to  the  Shift  and  Mask  Unit  is  rotated  by  an 
6 Ttic  Pm.ap,  tlie  Address  Mapping  Pi  ocossor  arbitrary  arr'our.t  and  then  masked  by  one  cf  3^  10-bd 

standard  musks  storod  .n  a PROM. 

The  mapping  prorrjssor  of  the  Kmap  or  P>map.  is  a 

Sixtenn-bil  horu’cntahy  microprogrjr.T.tul  processor  It  ^‘or  officirnt  addrr:,*,  mapping,  it  is  crucial  that  tlu'  Kn'/i,- 

occujups  a control  I'osition  with.n  tl  e knvip,  coorbinatmt)  have  fast  acc'^ss  to  the  udorniatiori  it  needs  to  perform  tlu* 

tin*  activities  of  the  other  comnomu  ls  it  is  PHH''  no{l  and  vii  tual-tc-phyr.ical  adrlrpus  translation  This  information 

has  a cycle  time  of  1 50  nanosccomis  V-  cromstructions  oro  consists  largely  of  the  active  Capabilities  and  soriment 

60  bits  Wide,  a !K*flO  bipol.ir  RAM  >.s  used  os  a writable  descriptors,  of  winch  up  to  4U0  n>ay  exist  in  the  cluster  at 

microstore  The  Pmqp  also  uses  a lugli-spocd  5K*16  RAM  to  d time  (sixteen  in  ejch  of  two  oudress  spaces  for  each  of 

sto»’e  the  active  Capabilities  and  segment  descnpto.-s  In  toiiriecn  processo'o).  Although  content  nddressah''^ 

•iildition  to  pcrfornuny  the  basic  addnvis  translation  for  the  nipmory  was  not  used  because  of  the  large  capnety 

ttonlocnl  references  of  a du-stcr.  the  ^iiuip  must  support  needed,  the  careful  poMfiomnvj  of  tables  within  the  data 

certain  operating  system  primitives,  statistics  gatlU'ruxj,  memory,  con'biiv'd  v/illi  a i-ash-coded  iist  structure  used  for 

H»\d  other  experimental  functions  without  excessive  storing  descriptors,  has  produced  n cacho-l'ko  structure 

perfornuinco  degradation 

The  data  mrmery.  or  t.Mata.  is  divided  mto  lOrd 
(expandable  to  •♦OPi.  ' records,  e.ich  recoul  containing  five 
1C -bit  words  ■^he  record  organ^’ation  was  chosen  I'eraus.* 
6.1  Data  Paths  the  sepi'u'nt  deser ->U  r-;.  v>r',tli  cache  ng  infofP'nbnn.  fi» 

coi'’fertfibtv  wdlun  this  aC-hit  spac<  t ach  wo'd  has 
A register  transfer  level  diagram  c'  the  Pmop  is  given  ossorintru  wdn  t t.vo  p edy  bits,  one  for  each  byte  The 

m Figure  6 1 Tlu*  ma  n da’o  paths  cons  st  of  three  mtcrr/ii  mef'iory  is  worrl  afldr  ''.vobic.  w th  t-'u*  record  ndc.'uss 

high  speed  tri-state  busses  Two  of  tlieso.  the  A and  B conun  j from  t'u*  DittA  «3(77ress  (D\pR)  and  Ihree-bd 
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Figure  6.1  Data  Paths  in  the  Pmap 


word  indices  from  fields  in  the  ciir'-ent  microinstruction, 
Tims  once  the  record  flciciross  of  a descriptor  or  cnpability 
luis  l)ccn  computed,  the  individual  siibwords  muy  be 
Mccossod  Without  expendmy  turVier  cycles  to  gonor/ito 
dotft  memory  addresses. 

D.itd  to  be  written  in  the  Mdntn  may  bo  taken  cither 
from  the  A Bus  or  E Bus  Beenuso  it  is  frequently  necessary 
to  set  and  clear  status  bits  in  segment  descriptors  (for 
exampio  the  "dirty  ” and  "use"  bits  used  for  demand  paying, 
and  tlu»  lock  bit  used  for  mutual  exclusion)  bit  set  and  dear 
logic  is  i>rovidcd  for  data  input  from  the  A Bus.  It  provides 
for  the  setting  or  clearing  of  eitlior  or  both  of  the  two  high- 
ordcr  bits  of  the  input  word.  To  further  mcreose  parnllohsm, 
it  IS  possiblo  to  simultaneously  read  and  write  different 
words  of  the  same  record  It  is  therefore  possible,  soy,  to 
set  the  "use  bit"  irt  one  word  of  a segment  descriptor  and 
at  tfio  same  time  extract  the  .segment  limit  from  another 
word  of  the  same  descriptor 


6.2  Micrcp'ogram  Sequencing  Logic 

One  cnarncfer.^tic  of  the  Cm*  addr<>.ss  mapping 
aUjorithms  is  the  large  number  of  conditions  to  bo  tested 


The  service  of  a typical  request  wiM  require  testing  of 
request  status,  operation  type,  and  .segment  type  and 
checking  of  the  following  conditions:  protection  violation, 
descriptor  locked,  segment  loctti-iabie  etc.  To  perform 
address  mappuiq  w thm  a reascnoblo  number  of  cycles 
require!  the  Pmap  to  have  a fioxiblo  mu'ti-way  branch 
cnpal)ility 

A block  diajr.i.n  of  the  aucroproyram  sequencing  logic  is 
givon  in  Kigufo  6 2 A ftaso  Jcfcfresi  i.s  selected  from  either 
the  Nftrt  /Jddress  f'O'J  n tiio  curror^t  microinstruction  or  the 
output  of  the  vSi/broiyf me  iinkajo  Registers  Two  bit.s  in 
the  microinstruction  select  the  mode  of  biancmny  (two 
way.  four-woy.  sixteen  way)  ana  two  threo-bil  fic'ds 
control  SIX  6-to-t  coiul  tion  code  mu'tiplexers.  W.iH  -Aay 
bf.Tnclung  wa.s  in>pic(Tionx**0  in  the  conventional  wav  by 
OR'inq  the  selected  conoition  codes  with  the  Base 
Address  T-u»  oddres.s  thus  gon»»rntud  »s  stored  In 
the  Microprog'i^m  ss  Register  to  fetch  the  next 

microin.stfuction  .s  a conthl«onal  Override  mechanism 

that  can  prunih't  a I'oteniiai  lb*way  branch  Wlien  the 
override  cond  t ou  'S  true,  a branch  Is  toxen  to  a 
sovenfeenth  loca’ion  rpjard-es.s  of  the  vaiue  of  the  I0*wa> 
branch  conuilion  codn 
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7 The  Line  and  interclu3ter  Bus  Structure 


6 Bus 


Ftgure  6.2  Microinstruction  Address  Generation  Logic 


6.3  Context  Considerations 


The  L*nc  provicJcs  mtercluster  communrcation  by 
connectirtQ  tho  Pinop  to  two  /nferefuster  busses 
CommumentJon  is  ni  the  form  of  short  messages  passed 
between  Kmaps.  Wessages  are  stored  in  a Message  RAM 
which  IS  shared  hetweert  the  Pmap  and  the  two  Interclustor 
Bus  Ports.  Pointers  to  messages  pass  through  an  automatic 
system  of  gueues  Messages  are  usually  sent  directly  from 
source  to  destination  cluster,  but  they  can  also  bo 
forwarded  by  intermediate  clusters  (thus  allowing  arbitrary 
network  topologies  to  bo  constructed).  Message  routing  is 
controlled  by  Pmap  microcode  The  goal  In  the  Line  design 
was  to  provide  fost,  deadiock-free  Interclusicr 
communication  wiU>  a nm-mum  of  Pmap  overhead. 


7.1  Intercluster  Bus  Protocol 

The  Intercluster  busses  contain  26  Imos.  16  data.  2 
parity,  and  0 control  They  operate  in  an  asynchronous, 
interlocked  fasnign  at  a transfer  rate  of  450  nanoseconds 
per  wo'-d.  Mastf*r-;n  p is  passed  cyclicly  between 
requesting  ports.  etft»ctively  implementing  a round  robin 
priority  scheme.  The  current  bus  master  arbitrates  tviture 
mastership  in  par.iHei  witii  Its  current  data  transfers 


Forword  Message 

15  1?  6 0 


c 

CX 

So'.iicp  1 Dc'.tinotion 

j 

OP 

J 

Otfnot 

Segment  Na*nc 

Data  Word  (VVrdn) 

L. 

J 

Return  Message 

15  1?  6 0 

r ■ ■ I N 

c|  CX  I in  1 1 1 jOGStinationl 
Data  Word  (Head)  , 


C Complex  Bit 
CX  Context 
OP  Op  Code 


There  are  a total  of  64  generni  purpose  and  32 
suhroutme  linkage  registers,  aiowing  each  context 
exclusive  use  of  eignt  general  purpose  registers  and  four 
subroutine  imk/igo  registers.  Tne  Cu'rent  Context  Number, 
stored  in  the  Confer?  Register,  selects  the  current  register 
hank  Normally  this  register  is  loaded  trC'm  the  Run  Cueue 
when  a context  swap  i.s  executed  For  diagnostic  piuposes 
the  Pm.ip  may  directly  lond  the  Context  Register,  hence  if 
rpguirref  n micronrogrnm  may  access  the  registers  of  any 
context  tnch  context  may  nest  sui>routine  calls  up  to  four 
levels  <lcep  By  convention,  the  ze-oth  linkage  register  Is 
also  used  to  store  the  reactivation  address  of  n suspended 
context  The  status  bits  in  the  Run  Oueue  indicate  whether 
n context  i5  to  bo  activated  at  its  reactivation  address  (to 
continue  on  ongoing  operation)  or  to  be  expi'citly  started  at 
one  of  the  first  sixteen  locations  m the  microstore  (to  begin 
a new  operation,  or  handle  certain  error  conditions). 


Figure  7.1  Standard  -\ic5sage  Formats 

intorri'jster  messages  consist  of  one  to  eight  16  bit 
words.  The  comn.ou  formats  are  sfiown  in  figure  ? 1 

The  header  word  contains  a six  bit  nloitifior  for  source  and 
rlcstmntion  cluster,  the  source  context  ni.moor  and  the 
complex  l)it  A return  n'essage  l»as  a unique  source  field  of 
all  ones  Tiu*  soiirce  context  nuribor  is  sent  w th  the 
mossaoe  to  aiiov/  a d.rcct  reactivation  of  tlie  susppnuod 
source  context  Tne  complex  bit  provides  an  escape 
mechanism  to  ot  'nr  r^es  jago  formats  eg  for  error  messages 
or  t)lcrk  transfers 
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Intercluster  Bus  0 


Intercluster  Bus  1 


Figure  7,2  Components  of  the  Line 


7.Z  Components  of  the  Line  (Figure  7.2) 

Buffer  space  for  messages  is  proviriod  In  the  central 
1K*18  Message  RAM,  divided  into  128  buffers  of  eight 
each.  This  is  sufficient  to  avoid  any  possibility  of 
clcudlocl;  over  buffer  allocation  except  In  very  largo 
systems  [Swan  ot  al,  197Cb]  The  Pniap  has  priority  for 
access  to  the  Message  RAM.  although  it  is  also  directly 
accessible  by  the  Ports.  Several  contexts  may  use  the 
Line  in  an  overlapped  fashion  without  mtor^eronce  since 
each  context  has  private  facilities  for  addressing  message 
buffers.  A context  has  two  ways  to  address  message 
buffers.  It  may  use  its  context  number  to  occess  a 
rasorvc^l  buffer  which  is  used  for  the  creation  of  forward 
messages  and  to  receive  return  messages.  There  is  also  a 
P/rtiip  Address  Register  for  each  context  to  deal  with 
incoming  forward  messages.  Words  within  a buffer  are 
selected  by  a Pmop  microcode  field.  Each  Port  section  lias 
an  n<t<lress  register  and  a word  count  rogislor  for  accessing 
the  Messago  RAM 

Five  queues  or©  maintained  by  the  line  Two  Sencf 
Oucn/cs,  one  for  each  Port,  are  used  oy  the  Pmop  to  request 


transmission  of  messages  To  request  that  a message  bo 
sent  on  on  Inlercluster  Dus,  the  Pmap  places  the  address  of 
the  message  buffer  in  the  appropriate  Send  Queue  The 
^ree  Queue  keeps  the  nddresses  of  oil  the  messoge  buffer.s 
not  currently  In  use.  The  Service  Queue  is  used  by  tl.o  I me 
to  notify  the  Kbus  and  Line  of  the  addresses  of  incominri 
forward  messages,  and  the  Return  Quoub  to  request  that 
the  Kbus  reactivate  contexts  when  replies  to  their  forward 
messages  are  received  Ail  of  the  queues  are  Implomenteu 
as  partitions  of  a single  1 K*  1 1 bipolar  RAM 

The  line  uses  the  some  150  nanosecond  clock  as  the 
Pmap-  f or  diognostic  purposes  tho  Pniop  has  ficcoss  to 
almost  all  of  the  Internal  stoto  of  the  Line  and  may  execute 
all  the  internal  microcycics  executable  by  tho  Ports 


7.3  Arv  li^tcrdustcr  Message  Transaction 

A complete  messoge  trarsf(?r  is  shown  m f igure  7 
Tiie  Pmop  at  the  source  cluster  creates  the  forweird 
message  m a rnsBrv»‘d  context  buffer  Then  its  pointer  («, 
put  into  tho  appropriate  Send  Queue  The  imc  pops  the 
pointer  off  the  Send  Queue  into  tho  Port  Adaress  Register, 
acquires  mastership  of  the  corresponding  bus  and  transfers 
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Figure  7.3  An  fntercluster  Message  Transaction 


tile  message,  one  word  at  a tune,  from  its  Message  RAM 
onto  the  Intercluster  Ous  and  into  the  Message  RAM  of  the 
clostinntion  Line 

At  the  destination  side  the  roco'vmg  Port  has  already 
ohtamed  a buffer  from  the  Free  Queue  If  the  message  Is 
received  completely  without  error,  then  its  pointer  is  placed 
into  the  Service  Queue  (if  not,  the  message  is  ignorod;  n 
timoout  will  occur  nt  the  source),  The  Service  Queue 
regucsts  tlie  Kbus  to  allocoto  a free  Pmnp  context  to 
service  the  message  it  inciueJes  status  bits  to  start  up 
specific  microcode.  The  context  wni  transfei"  tiu»  pointer 
from  the?  Service  Queue  into  tlie  Pmap  Addn?ss  Register  on:) 
prcicoss  the  message,  maxing  anpropn.ito  main-memory 
rcf*'rences  It  then  creates  a r»:*tiirn  nessage  in  the  same 
tnjffer,  setting  the  source  fioirj  to  ones  to  indicate  this  On 
u Read,  the  data  word  wii«  bo  appomied  Tlu»  buffer  pointer 
of  the  compictea  return  message  <s  queued  ogam  m the 
'■nd  Queue  vviien  the  message  has  hern  s^^nt.  the  pointer 
. released  into  tne  free  Otieuc  At  the  original  source  tl>e 
turn  message  is  placed  m trie  resinved  l)iiffer  *r>r  the 
’I'luesting  context  Its  context  number  plus  status  Is 
i>assod  to  the  Retuni  Queue  anti  tne  enntext  is  reactivated 
to  send  data  or  an  iick'nowledgcment  back  to  the  requesting 
proc ossor 


8 Dcvciopircnt  and  Diagncjstic  A»d.i 

A common  «tr.iti*gv  used  to  .Hi  m h.i'dware  and/or 
piK  rocorle  df' ve'0|>menf  's  to  constr./-.  t a software  stmuintor 
for  the  hardware  Tins  ni'ov/5  imt.ai  debugging  to  be 


performed  before  the  actual  hordv/arf*  is  avoilabie  and  can 
provide  a more  comfortahio  environment  in  which  to  work. 
However,  simulators  are  expensive  both  In  terms  of 
development  effort  and  computer  luno;  furihormo^c  tlury 
ennnot  give  an  exact  reflection  of  the  hordware  Thus  this 
fipprooch  loaves  the  final  bugs  to  bo  found  using  the  real 
hardware,  and  is  of  no  o'd  in  rtiagnOsSmq  component  failures 
(rather  tfian  design  nirors),  The  alternative  approoch 
arlopted  for  Cm*  was  to  incorporate  special  linrdwttre. 
called  //oof-.s.  d 'cctly  into  the  Knap  for  use  In  hardware  one) 
microcode  development.  Tiio  interfacing  of  the  Hooks  to  a 
standard  l.SI-lt  ai>nws  extensive  software  su)>port  for 
h.irfJware  dcveloi>ire  it  and  diagnostics  whiio  at  th('  sane* 
tune  provuhng  a convenient  environment  tor  the  deouqg  ng 
of  nncrocodo  on  the  roal  hardware 

TlU‘  Hooks  fjis/c  to  an  LSi-lt,  referred  to  ns  the  Hooks 
Processor.  t!u  obh-ty  to  mtui'atety  oxam.no  and  cl^nnge  the 
I'ltornal  state  o*  ti  c Kn»ap  Tiioy  provide  the  capability  fo' 
tlie  H 't'ks  f^'oeesser  to  load  imcroccKlo  into  th(*  wntaM' 
Contri  I store  of  the  ' read  tfie  valiu'S  on  thi'  A and  H 
l>tisses  of  Hv‘  P.iUiP.  und  to  iinicpendentiy  start,  slop,  an-; 
s:ngio-t  vrh'  Hu  P ’«up-l'nc  and  K»>ijs  rlnccs  An  uili  rrupt  e> 
ip'norateii  for  il.i*  Hoov  s Pujcessor  wiu?n(*v(’r  tru* 
oloc.l.  .stops  ^Oitiu'r  d ie  tf'  a uucropror;rani-.nvt)reu  tuUl  ( ’ n 
p’ornorv  f'li'd,  o'tor  on  tiu«  ccntri'l  or  data  strresl 
Furllurp’Of e s»’vef.u  o'  Hu  ritoTiai  reg'nters  o'  the  P'n.i.i 
have  "tw">  r('gist<;r t."  ussoc.atort  wH»  them  wiucti  may  ordy 
be  loadf  ) bv  tl»p  M i.  r-  Processor  These  niti**iMte 

rog  stors  n oy  bo  enab.od  via  the  H'^oks  to  overri.h' 
mir  roprof|',Tm  rnntr<.i.cri  v.ilues  Tnc*  presence  of  the  Mooxs 
adfied  <ipproxi'iiatelv  ten  pnreent  to  toe  cost  o*  tne  P»m<jp 
While  enorr-K'  iSiy  r 'duc:n<i  system  dctvc'opmcnf  time 
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9 Performance:  Measurements  and  Predictions 

Ooforo  di5CussH)Q  the  models  used  to  estimete  the 
performance  of  n Cm"  cluster,  several  simple  measurements 
(made  on  a cluster  containing  two  processors)  will  be 
presented  The  average  time  between  memory  references 
(including  both  code  and  data)  made  by  a single  LSI- 11 
executing  entirely  out  of  local  memory  vanes  between  2 5 
and  4.0  microseconds,  depending  on  the  mix  of  instructions 
being  executed.  For  a "typical"  code  sequence,  based  on 
measurements  of  compiled  BLISS- 11  programs,  the  inter- 
reference  time  was  30  microseconds.  Measurements 
made  on  the  same  "typical"  code  sequence.  exccf>t  with  all 
references  mapped  via  the  Kmap  to  the  other  processor  In 
the  duster,  yieidccf  nn  average  time  between  references  of 
r.7  microseconds.  With  the  latter  measuromont  there  was 
no  contention  for  use  of  the  Map  Bus.  Kmop.  or  destination 
Siocnl  Although  no  actual  measufemonls  wore  available  at 
the  tim€i  of  this  writing,  it  is  expected  that  the  tin-e  for 
interclustor  references  w»ll  be  between  15  and  20 
microseconds 

A Simple  queueing  model  was  deve-oped  to  estimate  the 
porf(»rmanco  of  a cluster  [Swan  et  , 197Gu]  Tho  model 
assunifut  an  exponential  distribution  of  nonlocal  requests, 
exponential  service  time  in  the  Pmop.  and  exponential 
distribution  of  the  total  non-Pmop  overhead  incurred  during 
a nonlocal  reference  It  Is  assumed  thot  tlie  Pmap  Is  the 
primury  cause  of  contention  hence  the  waiting  time  for 
other  facilities  is  ignored  Figure  9 1 plots  the  results  of 
this  analysis  The  relative  rale  of  memory  referencing  in  a 
c’ustor  IS  plotted  as  a function  of  the  number  of  active 
processors  and  theur  hit  rnf/o  to  local  n’emory 

Because  of  the  inability  of  the  (pioueing  analysis  to 
model  contention  for  all  cluster  facilities  it  was  fonred  that 
the  results  would  prove  to  bo  nn  optimistic  estimate  of 
cHislor  performonce  Therefore  a series  of  simulations  was 
pOfN>rmod  in  order  to  model  more  closely  the  true  operation 
of  n cluster  [Brown  1976]  Tne  simulation  and  queueing 
results  were  in  dose  agreement  and  so  the  simulation  study 
will  not  be  discussed  furttier. 

Figure  9 1 indicates  that  system  performance  is 
extremely  r1npcr>cient  on  the  local  hd  ratio.  It  has  been 
hypothesized  that  the  local  hd  ratio  woudl  lin  m tho  range 
bf  f veon  55%  and  95%.  In  whicfi  cose  tl>e  rff(*ct  of  tl>o 
nonlocal  references  wcnild  bo  "reasonably"  small. 
Unfortiinatoly.  this  imp'ios  thot  code  must  l>e  entirely  locol 
to  the  processor  executing  i!  F mc»'nor  y-mtensive 
prcxinims.  a quickr.oit  and  a memory  d-agnostK:.  have  l)ocn 
run  on  tl>e  initial  Cm"  system  (or^e  cluster,  two  n*ochiles). 
M'^asuroments  of  the  performance  dc.jrad ution  when  code 
and  local  vnnnblr's  are  kept  local  out  the  area  bemg  sorted 
or  di.if|nosod  is  moved  to  tho  other  processor  m the  cluster 
indicate  tfiat  local  hit  ratios  of  0C%  or  higher  am  bomq 
obtained  in  botli  caso.s  Expensive  operating  system 
functions  such  os  block  transfers  are  expocteri  to  lower 
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hit  Ratio  to  Local  Memory 


Local  Ri'tp'c'u-e  Ti.ne  = 30  uSec 
Lxlerndl  Service  Ti’iie  = 1 5 uScc 
plus  6 5 uSec  Constant  Ovor.hoad 

Figure  9.1  Absolute  Cluster  Performance 

this  figure,  but  it  is  also  expected  thot  most  user  programs 
Will  make  less  Mlensive  use  of  shared  databases  than  the 
above  oxam[)ios 

The  queuemg  model  was  used  to  prert>cf  tfic  aegrndaf  on 
of  cluster  performance  d eitlirr  the  Pmap  were  maor  siownr 
(and  thus  cheaper)  or  «f  the  concurrency  cf  tho  mopp-uq 
mechanism  were  eiiimnnted  Tho  results  for  a cluster 
containing  twelve  processors  are  shown  in  Figure  0 2 A 
slov/er  Pmap  was  modeled  by  incrcflsir.q  its  service  time 
from  1.5  to  3 0 microseconds  Ine  last  model  represents  a 
Cluster  implementation  where  each  external  reference  » 
corr.ed  to  completion  before  servicing  subsequent  requo'its 
This  would  he  the  sduation  if  only  one  Pmap  Context  weri' 
provided.  i e plimn\Htmq  the  concurrency  between  ttie  Map 
Bus  and  the  Pmap  Both  tho  slow  and  non-concurrent 
clusters  show  rnormour.  performonce  losses,  espoco'iy  at 
the  lo'w  en<J  of  the  ft9".  to  95%  hd  ratio  ranvje  Tho  innl>i!dN 
of  slov/er  or  non-conciirrfnt  Kmaps  to  support  inrqe  nup-ht'ts 
of  mcdulf’S  ffitpiics  a need  for  more  \nuips  per  Cm"  system 
it  also  suqqesta  tliat  more  ntcrclustor  communication  will  he 
required  su)ce  each  modulo  wll  have  fewer  immedtn»n 
neiqhhors 
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Hit  Ratio  to  Local  Memory 

Figiirc  9.^  Clu:itor  Performance  wttli  Slower  Pmap 
or  without  Concurrency  between  Pmap  and  Map  Bus 
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1 0 Conclusion 

Detflilod  hardware  dojigi  of  C ”*  bejm  » -n  laio  juiy 
1976  "''ne  tfiitiul  goal  of  a '0  (:roC(*ssnr  thrr-  c;ust<*r 

syston  IS  expected  to  be  res  /e  | .n  tee  fust  (ju.wtor  rf 
Considering  tin*  NftMn  He  ti'"e  from  tiif* 

Peginniny  of  design  to  a working  prototyue  (cxciviM  tie 
Line)  was  less  than  nine  motths.  it  is  fo  t that  th  s 
relatively  short  dcvolopmonl  tni’o  'S  tlu-"*  to  ettens  ./o  iix<»  nf 
nutcurMted  desifjn  aul  microprugr.ir*aiing  at  fihne.t  ♦•very 
level  «md  the  incKision  of  adddioiuil  hnr.j.varG  t > mri  m 
dehuggind  The  Hooks  fnciiity  m Uk*  k-mp  has  hc>  n 
[><'irticul,iriy  succer.r.fiil  HcK'/ever  \ >,  I nr't  be  po'.Ml.'c  to 
di'Clarf*  the  overall  system  n success  ,rU  ' d is  regulM''ly  ,m<J 
rel\.i)ly  sufJporting  a community  of  s.itish>'d  usc'3. 
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Abstract 

This  paper  describes  the  software  system  being 
cfeve/opoc/  for  Cm*,  a dlstribafed  muitt-microprocessor.  This 
software  provides  for  flexible,  yet  control'ed,  sharing  of 
code  and  data  via  a capability  addicssed  virtual  memory, 
creation  and  monayement  of  groups  of  processes  known  as 
task  forces,  and  officiont  interprocess  communication  Both 
the  softwore  and  hardware  are  currently  under  cofistruction 
at  Carnegio-Mellon  University. 
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Introduction 

Semiconductor  technology  advances  are  leading  toward 
the  inexpensive  production  of  computer  modules  (i  o a 
processor  plus  memory  of  a modnr.ito  si^e)  on  u single  dim 
Multiple  computer  modules  interconnected  to  form  u 
multiprocessor  or  a network  oVf*^r  a large  number  of 
processing  cycles  far  more  inexpensively  than  an  enuaily 
fast  uniprocessor  Vet.  such  a computer  module  system  is 
useful  only  if  a sudapm  fraction  of  the  processing  cycles 
can  actually  be  used  for  applications 

The  software  designed  to  manage  a computer  modulo 
system  can  contribute  sul>stantiaily  to  making  the  system  a 
cost  effective  onviionment  m which  to  program  applications 
This  paper  discusses  the  softwore  designed  to  manage  a 
computer  module  system  called  Cm*  which  is  currently  under 
construction  at  Cornegic-Mellon  University  Wo  ooy 
particular  attention  to  the  philosophy  of  software 
construction  tliat  influenced  many  of  the  design  decisions 

For  the  purposes  of  this  paper  we  will  only  review  some 
attributes  of  the  archdocture  that  are  salient  to  the  design 
of  operating  system  software.  Companion  papers  [Swan  et 
al.,  77a  Swan  et  al  77b]  describe  ana  discuss  the  Cm* 
architecture  m detail 

Cm*  is  a mu'tiproccssor  composed  of  computer  njod.jlos. 
each  consisting  of  o DEC  LSI-11,  a stondard  LSI- 11  l>us, 
p)c?mory  and  devices  We  describe  Cm*  ns  a multiprocessor 
because  the  system’s  pr'■n^^^y  memory  forms  a single  vutuai 
address  space;  any  processor  con  directly  access  memory 
anywhere  in  the  systen..  To  implement  such  a viMua’ 
memory,  we  introduced  into  eacl'  con*puter  module  n local 
switch,  the  Slccal^  which  routes  locoMy  gcnernlod 
references  selectively  to  local  moncry  or  to  the  Map  Bus 
(when  the  reference  is  to  memory  in  another  computer 
module).  The  Slocal  i.kcvvise  accepts  references  frorr 
distont  sources  to  its  locnl  memory 

Connected  to  a single  Map  Bus  nmy  bo  up  to  tourtei'M 
computer  modules  thcil  share  a single  address  mop;>mg  nml 
routing  processor,  called  the  .kmop  The  computer  module's 
Kmap.  ond  Map  Bus  togethor  comprise  a cluster  A Cm* 
configuration  can  be  grown  to  arbitrary  S'jc  b> 
interconnecting  clusters  via  Inter-cluster  Busses  (se** 
Figure  1)  (A  cluster  need  not  have  a direct  bus  con»>i''  l»on 
to  every  other  cluster  in  a contigurntion  ) CoHcct'vrly.  tiu’ 
Kmaps  mediate  each  nor'-local  reference  made  by  n 
computer  module,  thus  sustounng  the  ippenrnncc  of  a sinr^tr' 
virtual  address  space 

Because  processors  are  nuiierous  appluat-ons  o’  any- 
sire  will  tend  rot  to  be  designed  in  the  torn  of  o s'ngir* 
program  ex«»cuted  l»y  a sr'Oi^ntift  process  Instoarl  we 


Mn  Severn!  enses  names  of  Cm*  co'^ponents  ere  dCMvi*  1 
from  the  PMS  notation  ocscrib^'d  m [RcH  ano  Newell  71  ] 
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• Detail  of  a Computer  Module 
u 


(whtch  to  as  a user  hereafter)  is  confidont 

that  all  h(s  coeJo  is  dcDuogeo.  smee  lie  wiH  routinely  oiler 
poramotf»rs  and  evt*a  thn  code  tor  his  task  forces  in 
suhstontiQi  ways  We  also  expect  users  to  incrementally 
construct  experiments  In  addition  we  expect  users  to 
reconfigijro  mociiiles  (of  software)  comDinmg  them  to  form 
a now  experiment 

Such  a view  of  the  user  has  led  us  to  believe  that  it  is 
as  important  for  the  Kernel  (or  lowest  level)  software  to 
support  the  user's  software  ronstruefron  activities  as  It  is 
to  provide  the  pr-mitive  runtime  facilities  required  for 
multiple  users  to  sliare  the  computer  resources  In  a 
disciplined  cooperative  fashion  Consequently,  the  software 
design  reflects  this  concern.  We  view  users  as 
constructing  their  expennu'nts  by  incrementally  building 
modules^  Each  module  implements  some  abstraction  useful 
to  other  modules  that  will  come  to  depend  upon  it  A module 
then  IS  a 'unit  of  abstraction'  It  is  implemented  as 

--code  and  data  private  to  the  module. 

--a  set  of  externally  Known  functions  that  can  bo 
invoked  by  other  noduies  making  use  of  the 
abstractions,  and 

--a  set  of  references  to  externmiy  oef-ned  mudulrs 
defining  functions  used  in  implementing  thy 
abstraction 


The  kcrnul  software  supports  the  notion  of  a mjju  o by 
Figure  1.  A Simple  3 Cluster  Cm  System  provirtinq  user  faci' tips  to  crpn*e  nirduies  and  to  invoi-*' 

functions  of  a morluio  in  a proteerpu  way  An  mvov  r ; 


expect  users  to  create  task  forcos,  \ e groups  of  processes 
cooperating  to  achieve  o goal  Oecot/se  the  number  of 
processes  in  a task  force  may  vary  with  the  ava'inoir 
resources  and  task  paramoters.  anl  bocouse  processes 
tend  to  be  small  (duo  to  the  relatively  slow  processors  or 
Ifrnhations  on  the  amount  of  local  memory),  a user  w/H  often 
be  unconcerned  with  Imlividuoi  processes,  communicating 
only  with  the  task  forco  itself. 

The  Cm*  architecture  offers  to  a user  the  option  rf 
employing  tightly  or  loosely  coupled  processes  Loosely 
coupled  processes  communicate  rarely  usu.iMv  iri 
conventional  ways  via  a message  transmission  rnecnan’sm 
Tightly  couplod  processes  commumento  often.  srm<*t  mrs 
using  the  efficient  unconstrninod  paths  pf(»vi  h-d  b/  simrfni 
n cnriory  Cm*  permits  both  tyix  s cf  comnujuicntion  s'uce  d 
provides  a messntjo  transmission  fac'dy  os  well  a«-  dree? 
addressing  of  shared  memory  pifoctivo'v  a user  is  free  to 
view  Car*  as  e/ther  a aruifrprocessor  cr  a computer  network 


function  IS  t*xc.cutph  .u  an  environment  tluit  ci've*>  it  ac<  r* 

(c  coife  and  <laia  th.it  put  of  the  rnoui.»e  logetne'  w t*' 
any  artua  parnn- 'tprj  sp*':  fed  0>  the  'uoi  er  Thus  b *• 

software  ♦'n’o^ces  M'n  tv>  i.ui  »>  ^ , n*  a *i  m»  bv  = ■•  • » j 

a wi’ii  (l(‘*ineh  trar  •.  * * n lu*r  sei  ■»  »* » r itit'n  m nno  n 

and  pM‘Cul«on  m aar  f i<  ’ rh- ».  r,,.  , • . .v  hi  ; i '• 

the*  in^Ue  ire  of  »'r  or;-  av  ; <»*Pf  .f  I*  uen.ji  j 

This  no*  on  r -s  [ asi  h on  r‘%  <■>  w '• 

par»i.  .11(1'’  it  r»  hu  t • i !.'■**  : * h i a ili 

ii  scussetl  "1  [P.i'i’as  73]  and  at  st'C' ’ . «ta  tt  ••  , • 

7d]  ns  i»se<J  I'l  la*  i.iag  • oesxpi 

Madu'e  I'OaP  l.-i'  •>  are  .ised  tor  cre*e' t'on  purpr  ses  a’ 

runlu’u  ftuh  t i'  is  »‘»ecuti’d  vvt.i  mci'ss  ‘ , * 

tiu)  .<•  ubi»“-  * • whirh  r 'cg.j"'S  In  d'si'rung  tie  k--*'**' 
s-  '’w  'r»*  >ve  haw«*  f-  .nnj  that  s ■■■•*•  rf  its  »um’  . 

iPUPt-m*  nt  r itiK''  cc^P  I X (U-strnct'Ons  >nl  not  a uses  o* 
rt  modine  rc  iu'fo  t;u»  entxj  abstraction,  somo  uses 
on'v  on  pet  of  the  abstrartion  whi'e  others  re'v  on  a 
sinh»  f*»*  t nl'St'.u  ton  f o'"  design-'  ,>0’  • r^odule  f"av  h'* 


Software  Design  Methodology 

Cm*  15  a vehicle  for  cxpcr' •*e''tatiOri.  pir»r:ui  riv  m t'e 
nrea  of  parallel  decomposition  of  aihonthms  am  thex 
officicrst  implomentation  on  a con'ou’er  rr'-.duir  pr  . r ss";  j 
resource.  We  expect  it  to  bo  ra^e  thnt  an  p xpern«»»  ntr  r 


^t'lS  par  ■'  I'  wn,  . ‘isi  s the  wufd'  »r  rrou.j'e"  t 

ri'fi»r  til  thr  h.iM-v.ire  .trijcturp,  tmd  \v.  '.he  seg  ie'  use 
ti>e  ( or"rn,ie.»  ,»  . entcd)  S'ntpe  word  "e  <uie"  to  refer  to  a 
.s:  *.tf  .u  h:)n  Context  srjcuKi  also  so'’.'i»  t'- 

#•1  <•.  ••  i»i*  .»■'  > a'”  'I  , 1 ' » 
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partitioned  into  a strictly  ordered  set  of  /ouo/s  as  cfescriUeci 
in  [Haberinann  et  al  ?*C]  The  purpose  of  dividing  a 
module's  design  into  levels  is  to  permit  either  incremental 
introduction  of  the  difforrent  parts  of  one  abstraction  or 
increasingly  more  complex  (and  powerful)  versions  of  the 
entire  abstraction.  The  introduction  of  complexity  Is 
postponed  until  it  Is  truly  required.  Multiple  levels  of  one 
module  share  data  structures  and  even  code. 

The  first  level  within  a multi-level  module  may  define 
only  a subset  of  the  functions  of  the  complete  abstraction, 
but  that  subset  of  functions  is  a useful  self-contained,  but 
limited  version  of  the  abstraction.  Subsequent  levels  are 
introduced  into  the  hierarchy  as  needed.  Additional  levels 
of  a modulo  may  Introduce  entirely  new  data  structures  or 
extend  existing  ones  No  protection  boundaries  exist 
between  levels  so  that  higher  level  code  may  manipulate 
data  structures  introduced  in  lower  levels.  Consequently, 
though  module  boundaries  are  translated  into  runtime 
protection  boundaries,  the  boundaries  between  'levels  of 
design'  are  not  dotectablo  In  the  runtime  Implementation 
structures.  W«  will  illustrate  this  difference  between 
modules  and  levels  later  when  wc  discuss  the  Cm*  message 
tninsmission  module 

Levels  within  a module  are  strictly  ordered.  We  can 
define  a level  A to  1)0  'higher'  than  level  B in  another  module 
in  case  A invokes  a function  defined  in  0 The  set  of  all 
levels  (of  all  modules)  is  partially  ordered  by  dependency 
In  tiu'  design  of  operating  system  software  there  is  not 
necessarily  a cleanly  idcntifial)lo  div<sion  of  a hierarchy  of 
levels  into  supervisory  and  user  software.  The  operating 
system  facilities  rocimred  by  one  user  differ  from  tl)03O 
reciuired  by  another,  particularly  in  an  experimental  setting. 
The  partially  ordered  .system  structure  is  in  a form  such  that 
it  IS  n?ad. ly  possible  to  replace  ’ui>i>cr’  portions  of  the 
dependency  hierarchy  since  level  buiindncs  are  dear  and 
the  (lepentlency  relations  between  levels  arc  known. 


Cm*  Software  System  Design 

Before  describing  the  kernel  software  design,  wo  will 
defjni*  two  notions  that  play  <»n  important  part  in  that 
di'vqn  objects  and  capability  aiJdri'ssing  of  objects  The 
basic  unit  wlvch  can  be  namc<l.  shared  and  Individually 
protected,  and  for  winch  momery  »s  allocated  for 
representation  purposes  is  tlie  cbfcct  Fnch  object  has  a 
uMKpic  name  and  a dof.ntivo  description  used  by  the 
S'  ffwa.e  system  Every  object  hos  a type  that  determines 
the  structure  of  its  representation  and  the  operations  or 
ar  esses  which  can  bo  perfor-ned  nn  it  Current  design 
sf)i.-c»f'Os  three  types  of  objects  duM  sogrnenti.  which  ore 
•iicir  arrays  of  words  that  may  bo  road  and  writti'n. 
• ap.i/i'bfy  fir.ti,  v/luch  are  structures  containing  capabilities 
(to  1)0  discussed  bciov/).  and  m.ii/bo*cs.  which  are 
structures  containing  messages 


Objects  are  named  (addressed)  using  capAbihties 
[Denius  and  Von  Horn  C8.  Lampson  G9].  A capability  may 
only  be  created  and  munipuiated  in  controlled  ways  (by 
kernel  provided  capability  functions).  Since  users  cannot 
create  or  forge  capabilities,  possession  of  a capability  Is 
evidence  that  the  user  can  reference  the  object  whose 
unique  name  appears  within  the  capability  A capability  not 
only  identifies  a unique  object,  it  records  a set  of  rights 
indicating  which  of  the  defined  operations  (accesses)  are 
permitted  to  bo  performed  on  the  object.  Controlled  use  of 
objects  is  enforced  because  an  object  can  be  accessed 
only  if  a program  presents  a capability  naming  that  object 
which  contains  a right  for  the  desired  access  Since 
possession  of  a capability  endows  the  possessor  with  the 
ability  to  perform  accesses,  capabilities  also  record  those 
rights  which  a possessor  may  exercise  with  respect  to  the 
capabilities  themselves  (for  example,  copying  a particular 
capability  may  not  be  permitted.) 


Based  on  the  above  discussion,  we  next  describe  the 
Cm"  kerne!  software  The  puri.'ose  of  the  initial  levels  of 
software  is  to  provide  fauMics  recjuircd  for  shared  usage 
of  resources  in  an  'enforcabiy  cooperative'  way  In  addition 
we  wish  to  assist  users  m p.-ogramming  and  t.-xccutuuj  theur 
experipients  by  provuiiiig  convenient  structuri?s  end 
functions  for  creating  and  executing  modules  The 
operating  system  sottv/aro  itself  is  composed  of  a partially 
ordered  set  of  levels  In  several  Instonces  two  modules  are 
divided  into  a pa.r  of  levels  For  convenient  reference 
levels  are  labeled  with  a tag  in  the  format  'mcdLilp-levot' 
Modules  ore  given  nipnabetic  names,  -ovels  arc  numbered  in 
increasing  order  as  they  oppear  m the  system  constriict'on 
hierarchy  Ti»e  kernel  levels  to  bo  O scussod  In  this  j>aror 
ore 


CAP-1  Capal)ii'ty  referencing  Performs  rroppiitq 
from  n capability  vm  a segment  descriptor  to 
piiysical  f ep'’fsentntion  of  segment  (including 
access  control  checking) 


CAP-p-  Cnji-ibii'ty  addressing  and  mer»io»y 
allocation  ncfinns  nn  object  address  soaro 
and  mterprftnt.on  of  on  address,  perforn-s 
memory  allocation  erisunug  th.ut  tlie  segments 
used  to  repri  sunt  objects  are  pairwise 
exclusive 


Mr-1  fnviionmonts  n*  d Modules  Inplpmunts  the 
croaticm  and  deletion  of  modules  and 
execution  f'nvuonnflnts 


MSG-1  Cond  t cn.ll  message  trarisnussinn  Oefini's 
tito  structures  m«'ssagp  and  rnalhox.  permits 
send.ng  and  reccivog  of  messages  when 
process  suspensicn  -s  not  required 


DSP  D'Sj).itr^  ng  Dcfi  ii's  hardvvore  Iniplementod 
data  strui  turns  used  to  load’  an  eivnonmcnt 
onto  the  processor  and  commence  uxecution 
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MPX;  Multiplexing  Selects  the  next 
to  execute  on  a processor 


onvnonmcnt 


ME”2  environment  relations  Records  the  ancestry 
by  which  environments  are  related;  provides 
for  nested  and  parallel  execution  of 
environments 


MSG'2:  Unconditional  message  transmission 

Provides  for  sending,  receiving,  and  replying 
to  messages  even  If  environments  involved 
are  forced  to  wait  for  on  indeterminate  time  to 
complete  message  transmission 


Tl:  Trap  and  interrupt  handling  Provides  routing  of 
control  wiien  either  interrupts  or  traps  occur 

A diaqr/im  indicating  tlie  dependency  relations  among 
these  levels  appears  ns  Figure  2.  An  arrow  from  level  A to 
level  3 indicates  that  a funtion  m level  3 Is  invoked  »n  level 
A In  nddition,  it  is  possible  tlhit  level  A invokes  functions  in 
any  of  the  levels  'below'  3 in  the  dependency  graph. 

Capability  Addressing 

Module  CAP  provides  capability  addroGsmg.  Level  CAP- 
1.  which  IS  implomented  in  Kmup  microcode,  interprets 
capability  references  to  objects,  i e.  it  maps  a capability  to 
the  pliysical  representation  of  the  object  named  by  the 
cnpnbiiity  Because  the  state  of  an  object  may  change  and 
its  physical  representation  may  move,  tnn  system  nianita'ns 
<1  singlci  (lofmitive  description  of  each  object  caMcd  fl 
cf>^uc.nptor  or  descr/pfor.  It  records  the  type  of 

tile  object,  the  physical  itt.'scnptinn  of  its  representation 
(inchniing  cluster,  module,  starting  add'ess.  and  sue),  state 
ii>formation  (e.g  whotfier  the  rei>ror>entntion  is  in  core,  nirty, 
or  locat'd  for  Kmap  usage),  and  the  (reference)  count  of  the 
numbo’’  of  outstanding  copabi-ities  for  tiio  objoct. 


fcvery  existing  Object  has  a un-gue  nHme--the  m*'mcry 
address  of  its  descriptor,  To  perform  a mapping  from  a 
cnp«d)iiity  to  a object,  the  identity  of  the  object's 
descriptor  is  eJetormmed  from  the  cupabi  ity  It,  in  turn.  Is 
referenced  to  deto.'mme  the  physical  representation  of  the 
objr-Ct  A capability  reference  f.uis  if  the  right  reguend  to 
|)L'rforin  the  operation  desT'Ml  by  the  addressing 
onvi'onment  ongimiting  the  refc'enco  Is  not  In  the 
copnt)iiity 

level  CAP-r  oxtends  h'vel  CAP- 1 to  provirt,**  for  the 
rjrner.jtion  of  capability  ri)fer«,'ncus  refer  to  this  as 

copabihty  atidressing).  ana  for  can.ib-  ty  manipulation 
Cupaoil.ties  used  ^or  addressing  puriioscs  arc  stored  in 
capability  array  Objects  called  caruhi//fy  //sf5.  Given  a 
copaluhtv  hst  CL  and  an  index  X.  one  can  doterminu  the  X* 
th  capability  in  capability  list  CL  Tms  may  be  a capability 
for  an  object  of  arbitrary  typo.  incUid'ug  n capability  list 
object  By  repeated  application  of  capabi-ty  indexing. 


Figure  2,  Levels  and  Modules  of  Cm*  Softwai’c 

objnct.s  to  any  dc(>th  can  be  aUdrcssc'l  (ioenuse  cnpnbiiity 
list  in<lcxuKj  is  pcMoriUOU  in  imcrocoue  as  weM  as  in 
software,  the  arch.itecture  restricts  ingcximj  to  depth  ? ‘n 
any  S'ikjIo  operation  T us  .neans  that  m a sing'e  addressrig 
operation  tho  path  to  a target  object  may  'indirect  through' 
at  most  two  capaoi'ilv  lists  before  arriving  at  the  (ihf»'d) 
target  object.  Whenever  a processor  ts  executing  (,e 
generating  capability  afldrr'-^ros)  one  capabd'ty  hst  is 
d stmguishod  as  tne  p'/mdry  capabif/fy  hst  The  fest  index 
of  a cnpabilty  address  is  an  offset  into  this  pn-iav 
capnbii'ty  1st 

CAP-2  a'Sf>  dpf' >e-t  fn  iC'ocodeci)  finrtions  h\t  createnj 
copvng.  and  dcet-ij  canae  dies  we*i  as  h i 

nwinip  u.itm  ) the  ri  jv.'i  merch  <1  v..t'''n  a mpar  -dy 
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A Cm*  processor  (on  LSI-1  1 ) has  a word  sue  of  only  16 
bits  To  permit  16  bit  addresses  to  be  mapped  to  the 
arbitrarily  sized  Cm*  memory,  the  notion  of  a w/nOow  was 
introduced.  It  consists  of  15  iv/ndow  registers,  each  of 
whtcti  can  be  thought  of  as  holding  a capability.  (Actually, 
in  the  current  design,  each  window  register  holds  an  Index 
to  a capability  which  can  be  indexed  via  the  current  primary 
capability  list.)  CAP-2  provides  two  (microcode 
inipiomented)  functions  SegfoaO  and  Unload  to  associate 
and  de-assoclate,  a window  register  and  a capability.  To 
read  or  write  a data  segment,  a capability  for  the  segment 
must  be  Seg/oaOed  into  a window  register. 

A 16  hit  machine  address  is  Interpreted  to  select  a 
window  register  (and  thus  a capability)  and  possibly  to 
specify  un  offset  into  a segment  of  memory  For  enhanced 
performance  of  capability  referencing,  the  descriptors  for 
the  objects  named  m the  capabilities  associated  with  the 
window  registers  are  coched  in  the  Kmap.  This  mechanism 
provides  Virtual  addressing  and  allov/s  for  conventional 
relocation  of  physical  memory.  It  is  sufficiently  general  to 
support  the  definition  of  Kmap  microcodod  operations  on 
capability  lists  and  mailboxes. 

The  lost  facility  introduced  in  CAP-2  is  that  of  memory 
allocation.  Physical  memory  is  allocated  to  hold  segments 
so  that  no  two  seaments  overlap. 

Modules  and  Environments 

Level  MC'1  provides  for  the  creation  and  deletion  of 
modules  (as  discussed  earlier)  and  for  executing  invoked 
functions  A module  is  implemented  by  a module  cepabitity 
list  containing 

--capabilities  for  the  code  and  data  segments 
fcguircd  to  perform  the  functions  defined  in  this 
mndulc. 

--a  data  sorjnc'nt  containing  a vector  of  function 
descriptors  wiucii  specify  the  codt'  to  be  executed 
when  a particular  function  is  invoked  (e  g the  Index 
into  the  module  capability  list  for  the  segment 
containi'Kj  rode  for  this  function),  the  number  of 
parameters  expected  and  tl-.e  sue  of  stack  required 
to  perform  the  function, 

--a  hst  of  other  *knov/n'  nodules  containing  functions 
that  can  bo  invoked  by  this  module 

MT.-1  also  defines  on  environment,  the  structure  created 
ns  a result  of  a function  invocation  An  environment  Is 
defined  by  several  objects,  one  is  the  primary  capability  list 
wluch  IS  private  to  a function  invocation  and  acts  as  the 
root  capability  list  for  all  addressing  of  objects  during 
execution  of  the  function 

The  pri'twiry  capability  list  contains  capabilities  for 

--the  execution  stack  (private  to  the 
eovironment) 


— the  module  capability  list  which  defines  the  module 
containing  the  invoked  function, 

--a  sfofe  uecfor  (private  to  the  environment)  which 
contains  the  processor  and  addressing  state  wtien 
the  environment  is  not  executing  on  a processor 
(The  state  vector  includes  processor  registers, 
processor  status  word,  scheduling  data,  trap  and 
error  masks  for  communicating  with  the  Kmap,  and 
indices  of  the  capabilities  Seg/oaded  Into  the  window 
registers  during  the  environments  execution.) 
--parameter  objects  specified  by  the  invoker 

The  module  capability  list  contains  capabilities  for  those 
objects  shared  by  ail  who  invoke  a function  in  the  module. 
The  primary  capability  list  contains  capabilities  which  are 
local  to  a particular  invocation  of  a function 

Level  MEi-1  provides  functions  for  the  creation 
initialization  and  deletion  of  modules  and  environments 
These,  in  turn,  are  used  by  level  MC-2  lu  providing  functions 
reiating  the  execution  of  different  environments.  Functions 
Call  and  S^eturn  uUow  nested  execution,  I e the  Ca//i  uj 
enviionment  is  suspended  for  tiie  duration  of  the  execution 
of  the  newly  created  (Ca//ed)  environment  which 

terminates  when  the  Ca/fed  environment  Returrn^  The 
function  Fork  permits  on  enviionment  to  request  thot  a 
function  be  invoked  to  execute  iii  parallel  w th  its  mveker 
until  the  function  Jo:n  is  performed 

ME-2  imtidluos  a nrwiy  created  enviionment  to  r(*cord 
priority  I'lformation  for  scheduling  purj>oses  and  to  record 
the  existence  of  a ncwiy  created  onvnonment  in  tl»o 
Hneegv  (tnnuly  treu)  of  ds  creator  IT  is  Uiis  linoa'io  wiw  h 
IS  used  by  still  higincr  levels  to  keep  truck  of  a task  force, 
the  set  of  onvuonments  which  am  coopemting  to  achievi’ 
some  goal 

Message  Transmission 

The  members  of  a tasu  force  need  to  be  able  to 
synchronize  their  actions  and  to  con'municatr  with  one 
anotlier  To  this  pr^d  moduU?  MSG  defines  an  abstractio.n  o' 
a mailbox  winch  con  contain  messages  A ma<ii>ox  -s 
capablo  of  containing  fixen  finite  Mimbcr  ( f nH'sHiig*':. 

maintained  m FdO  oruor  To  permit  u.ofs  to  comiruniLatc 
arbitrary  objects  to  enu  anotlmr.  mti  er  tiian  data  only, 
messages  arc  jjairs  of  co.'abi'ities  (To  tmnsr  t 16  bus  of 
information,  a iis(t  can  crr«tf  a nata  to  contain 

tins  user  specified  infum  ulion  ) 

levels  MtiG-1  and  V.<G*2  d ffor  in  fl  at  MSJ-I  prov«rirs 
only  the  functions  Coud.Si'nj  and  ConaRv  e.w  to  tiafS»’M 
messages  wlufn  thesr*  funt.tions  can  hr  co'^'pirte  i witl> mt 
SUSpri>SiOfi  of  tin*  invoker  CoacfS-  n>!  surceeif-;  » 
depositi'Uj  a moss.u.e  nto  a r"tj'*box  on.y  «f  *1  e ma-ifsox  bns 
rcom  for  it  Cor}UR<'i,e  ve  s a function  whu  rr*tu'ns  ti  ** 
oldest  mesbngc  m I'ase  the  notUoK,  s not  f'*'pty  Henci' 
CondHecoive  can  oo  use.)  'or  noi'ie.g  A rcre  ve1  mrssaus  \ 
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placed  i!i  the  recoivnig  environment's  mess<J:pe-poi/ch.  a 
dcsic|ri«tecj  pair  of  positions  in  the  envuonment's  primary 
capability  list  ConrJScnd  and  Cor^dRecei^e  will  return  an 
error  code  if  the  mailbox  overflows  (is  full)  or  underflows  (Is 
empty),  respectively. 

The  second  level,  MSG-2,  extends  the  set  of  message 
transmission  functions  to  provide  a synchronization  as  well 
as  a communication  mechanism.  MSG-2  is  relies  on  the 
hierarchy  above  the  MPX  level  where  the  notion  of  blocked 
environmonts  was  introduced.  MSG-2  provides  the 
unconditional  message  functions;  Send,  /?ecG^ve.  and  Reply 
Send  perforins  the  same  tasks  as  CondSend;  except  when 
the  target  mailbox  Is  full.  Send  vrill  cause  the  sending 
environment  to  be  blocked  owaiting  an  opportunity  to  deliver 
Its  niessogo.  Likewise,  the  Recon/e  function  couses  the 
environment  attempting  to  Rocetvo  a message  from  an 
empty  mailbox  to  become  blocked  SenOing  a message  to 
nn  er^ipty  mailbox  on  which  an  environment  is  wailmg  will 
cause  that  envnonment  to  Recoivo  the  message  ond 
become  unblocked.  Sinnla'ly,  if  Receive  causes  a full 
mj-lbo*  to  no  longer  be  full,  it  will  awaken  the  oldest 
environment  awading  to  deposit  a message 

MSG-2  also  defines  a Reply  function  for  mailboxes  This 
function  differs  from  Send  in  that  after  executing  the  Reply 
function  on  a moilbox  as  permitted  by  a capability  for  that 
naiibox,  the  ngnt  to  Reply  to  that  mailbox  is  removed  from 
the  cnpabi’ity 

The  two  levels  of  the  mossago  fr«insmission  module 
provide  an  excellent  example  of  n decomposition  of  a snglo 
module.  MSG-1  defines  both  message  and  mailbox  <lata 
structures,  but  provides  functions  which  are  of  himtod 
applic  ability,  m some  sduations  the  functions  ♦nil  rotiiriiir)q 
an  err<>r  code  Conditional  functions  are  used  to  transmit 
me*.srt<irs  in  a well-doftnod  fasnion,  l)ut  <lo  not  perform 

syn  •hrii'U/ntion 

M.SG-2  extends  the  definition  of  the  mailbox  data 
strticture  so  that  waiting  enviionmcnts  con  be  recorded 
wlu'n  necoitsary  It  also  provmes  new  functions  extending 
the  usefulness  of  mo'iboxcs.  but  not  ‘covering  up’  or 
subsuming  the  cond.t'Onul  functions  wmeh  arc  useful  when 
polling  IS  desired  The  muit.plcx.ng  module  rel'cs  on  the 
conditional  message  ftinctions  of  MSG-1  and  impltmunts 
b bckiiu)  ond  unblocking  on  which  tno  second  level  of  MSG 

rtepenfis 

Dlspatcbinj  and  Pyiuitiplcxing 

D >pafchmo  fOSP)  nnd  Maitip^oxinn  (MPV)  are  both 
tr-vels  arul  enti'e  modules  OSb  ticfi'ins  the  hardwam 
iinplptm*ntpd  state  vector  and  .ts  associated  tnviOyid 
tuf>'':tion  wh'ch  loads  an  environment  onto  a computer  module 
nnd  bf?(|inn  execution  fnvfoart  .s  imprmenfed  in  a 
comb'll. ition  of  Kmop  microcode  and  .sc*t-Aare  Software 
portions  of  FnvfoPd  locoto  the  process  register  values  and 


the  processor  status  word  values  m the  slate  vector  and 
load  them  into  the  ptiysical  processor  registers  The 
software  then  stores  the  index  of  its  capability  for  the 
environment  m a special  location  which  alerts  the  Kmap  that 
an  fnv/oad  is  in  progress.  Tite  Kmop  portion  of  this  function 
loads  appropriate  values  found  in  the  state  vector  into  the 
window  registers  ond  various  Slocal  registers 

Functions  in  OSP  are  used  exclusively  by  tne 
multiplexing  module  (MP.X)  which  is  responsible  for  selecting 
the  next  enviror^ment  to  be  EnvtoaOeP  Module  MPX  defines 
a set  of  Runquoues.  each  of  which  is  a mailbox.  If  an 
environment  is  eligible  for  execution,  i e.  It  is  not  blocked 
nor  already  executing  on  some  processor,  then  there  Is  a 
message  containing  a capebddy  for  It  in  one  of  tho 
runqueues 

Associated  with  each  processor  is  an  ordered  list  of  at 
least  some  of  the  runquoues  The  orclorjug  selects  the 
priority  with  winch  that  procr'ssor  services  the  mailboxes 
The  same  Runquoue  mjv  appear  m various  positions  in  the 
ordered  l'5t  of  runriueues  of  odfe  ont  processors  The 
Multiptet  function,  invoked  hy  t/ie  superior  levels  MC-2  and 
Tl,  cycles  dov/n  the  lut  of  funquet.es  (private  to  th.p 
processor  oxocutmg  A?u/t/pie»)  performing  CondReceivos  on 
tiu*  runtpicues  if  tho  Con(;Re*:e/vu  'S  successful.  tlu»n  the 
result  ».s  H cftpsb.i'ty  for  the  next  env*rt>nmou!  to  be 
EnvIondciS  on  the  exociiti  ig  processor 

Trap  and  Interrupt  Handi  ng 

Software  traps  ac  I I'lterrupts  signal  exceptional 
conditions  caused  by  program  action  and  extornn' 
asyncfir^Tnous  events,  respectively  With  only  a few 
c'xccplion.s  (i-' q responding  to  fl  dock  interfutt  or  to  a Iv.n 
speed  device  interrupt).  hni'Uware  traps  and  interrupts  pro 
translated  into  software  Imps  and  interrupts,  so  U»at 
modules  can  indicate  wluit  action  is  to  be  taken  when  tlu?y 
occur 

Defining  a ne-.v  trap  (intT*m,pt)  means  defining  a nc.v 
IfAp  { inter  r jpt\  i/cctor  cntfi  mdicotiiKj  what  fufition  in  whnt 
modulo  IS  to  be  invoi-ed  if  the  trap  (mterrupl ) occurs.  Wui'n 
a trap  occu»s.  it  was  cause')  oy  the  executing  environment 
so  a C'i'l  IS  pcrf(»mc')  to  s ispomi  the  current  oiwiiormrnt 
and  cmis^  ti^e  function  named  n the  nppropn.atu  trap  s*et  tor 
€'ntrv  to  be  e*ccuieJ 

Interrupts  are  as /iichronous  and  are  not  necessar 
rehited  to  the  cuiri»nt  I>'oc“^sor  enevutir»M  Ti  of tw 
options  As  a resi..t  of  nn  inlcrri.pt  a fc'K  can  be  prrforru'tl 
to  the  function  nami'cf  *n  the  assoc  -j'ed  interrupt  vector 
This  W'll  c ause  the  •ntc*'’f‘ipt  to  be  sr'r-.cc*  j m parni.e  w f’ 
evccut'OM  of  oJiier  r-iiv.  iw.-^v'nts  Ad  e*  M.it,vcly . ar^  intern.,  ♦ 
vect^ir  cr  tr.ie  v«»cto»  •*>  rrnv  dire  t *i'..'  as  a resu  t ■ * . 
idtrrn.pt  status  n*or’^^\cn  be  sent  as  a messaun  f'  ■. 
spec'f  ed  r-.i.n  ..*  Prcsi.ma:'  y some  on v.r onm,.*s»  , ar’al-  » * 

finno*  j ?hi»  nt  er  rui  I w e * « • or  c nn  jRet  we  >.  • h • t ' • 
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message.  Interrupts  would  then  be  procc»ssed  seguent»ally 
by  order  of  occurence 

Two  observaions  are  appropriate  here  On  is  that  using 
the  trap  ond  interrupt  mechanism,  any  level  above  Tl  can 
define  vector  entries  so  that  code  from  higher  levels  can 
respond  to  exceptional  conditions  encountered  when  code 
from  lower  levels  is  executing.  This  effects  'outward  colls’ 
so  that  lower  levels  can  rely  on  higher  levels  when 
exceptional  conditions  anse.  The  second  observation  is 
that  the  trap  and  interrupt  module  is  quite  small,  relying 
heavily  on  ME  tor  Fork  and  Ca//,  and  on  MSG  for  mailboxes. 


The  Kernel  System 

The  Cm*  architecture  provides  alternative  ways  to 

implement  functions.  A function  may  be  implemented  In 
Kmap  microcode,  or  »t  may  be  implemented  in  softwa-e  to  be 
executed  by  one  or  more  of  the  computer  modules.  A 
computer  module  may  execute  a function  in  either  of  two 
address  spaces  (user  or  kernel  space)  The  decision  where 
to  place  a particular  function  of  a parliciuar  level  of 

a particular  module  is  determined  1)/  considerations  sucf) 
ns  maximizing  performance.  providing  for  proper 
synchronization,  and  ease  of  implementation,  as  well  as 
maintaining  protection  boundaries  bctv;een  module. 

Because  of  this  indcpcn<lonce  between  the  design  and  the 
physical  realization,  alternative  miplomentalinns  of  a 
function  are  possible.  This  facility  is  expected  to  be 
valuable  In  a system  designed  for  experimental  use 
because  it  allows  for  function  substitution  and  redesign 

Tlx*  kernel  software  system  described  here  is 

implemented  in  two  parts:  Kmap  microcode  and  a set  of 

programs  wl>ich  run  in  the  kernel  space  of  the  computer 
mod. lie  processors.  It  is  intended  that  m the  uiitial  system 
all  of  the  capability  functions  and  message  functions  will  be 
performed  by  Kmap  microcode  The  ren.aimng  functions  will 
PC  mptemented  m software  to  be  executed  from  the  kernel 
space  of  the  computer  modules. 

The  kernel  and  user  spaces  have  symmetric  data 
structures  because  both  are  oxccutriig  envnonmc’its.  fioth 
the  user  and  the  kornol  s>slem  have  a primary  capability 
list  which  acts  «s  a ’root*  Tcr  capability  addressing 
purposes  Moth  prh'iary  capabit'ty  lists  include  a copahinty 
for  a state  vector  and  for  a mcduic  capability  list  it  »s  tno 
prmuiry  capability  hst  and  the  state  vector  of  the  kcronol 
space  t''fH  maintain  information  pfirticul.ir  to  a proressor 
Shar«»d  data  and  code  m the  xcmei  a'O  referenced  via 
capabilities  in  the  kernel's  meduie  copab,  ty  hst 


Status  of  Software  Development 

As  of  December  1976.  the  microcode  available  provided 
only  for  simple  relocation  of  physical  addresses  with  no 
capability  referencing  Development  of  microcode  to 
support  capability  operations  and  the  message  facility  will 
follow  shortly 

Kernel  space  programs  have  been  coded  in  BUSS-11 
[Wulf  et  Qi  71],  a system  implementation  language  Tlus 
set  of  programs  is  being  tested  using  a simulator  for  the 
Cm*  macinno  [Chansicr  70]  winch  executes  on  C.mmp. 
anotlier  multiprocessor  system  developed  at  Carnegic- 
MoHoii  University  [Wulf  et  al.  74],  The  simulator  mode's 
multiple  computer  modules  as  multiple  processes,  and  is  able 
to  run  at  about  half  the  speed  of  a Cm*  processor  by 
explnitiirj  the  vvr.iablf;  control  store  features  of  the  C rnmp 
mu'tiprocessor.  Since  the  kernel  code  is  successfi.i'v 
executing  on  the  si'^ulator.  it  is  expected  that  the  softwart' 
kernel  will  bo  available  for  use  shortly  after  the  completion 
of  the  Kmap  mcrocoumg 

Future  Softwore  Development 

The  kernel  system  mcduics  as  described  constitute  a 
very  primitive  system  A number  of  additional  software 
levels  and  new  modules  are  in  various  stages  of  design  it 
IS  expected  that  most  of  the  levels  in  these  modules  wH  be 
implemented  as  programs  in  the  user  space  Modules  under 
development  Include. 

Secondary  Store  f.1'inaqcmcnt--Currcnt  design  proposr'.s 
ockliruj  some  ciisK  memory  local  to  sonu;  clusters,  w.lh  larfjo 
flic  storage  accessible  via  a h-gh  speed  link  to  either  the 
C.mmp  or  the  DEf  KL-1  0. 

Linkeditinq* ' Tho  creoti‘''n  and  management  of  moiluic''*  n-« 

Cm*  modules  wll  h(»  performed  by  o i nkeditor  mtenderi  to 
simplify  the  constnivtion  and  man.ujenient  of  functK)n 
tables,  segau’ots  of  code,  and  mvocat.on  serjuencos 

Command  lnterpreter--This  mcduic:  wHI  provide  on-lmc 
Interactive  access  to  the  Cm*  machine  Tius  will  ailrw  a 
prorpammer  to  dyi'.amicai'y  n\nage  a task  force  Cmri'ut  , 
interactive  terminal  comn.jnication  is  p'ovided  by  a PDP'1  1 
connected  to  each  con  ['utcr  n odule  by  a serial  i-ne  u»ui 
[Van  Zoe-en  76] 

ALGOL  f'M  rtunti’iu*  Systeni-“Tnr  first  pf('gr/»r 'U  m.} 
systeni  to  be  ava  aoic  cn  the  Ciu*  mnc’hne  is  expect  *d  tr 
bo  Al  GOl  flS  (Unti'  sudi  a ‘‘vsten  is  nvi..lat''r,  crop  w ' 
cross-conp'Ied  on  anolln’r  CMChme)  Tins  version  of  /^l  •*'>  ' 

03  w.il  I'p  desiTH'  i to  cxp'O'l  the  Miuit,proci's;;ing  fa  .iiti.’-. 
of  the  Cm*  fTiAcn  ne 

fb'SOiirCi'  b.v.v  r • iiTi  s -A  tfi:.*-  * M‘  iim**’.  r*  a f 

rur^time  'P*  CO'H  »*:oinu  sc'  i Ju  n J and  fpsi-.n.  *' 

al'CCQtion  tt  *s  th'  tn*.-  * « PC'C>  nu  o to  » ■ 
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these  ()oc»s>ons  hasecJ  up  on  tiie  hynumic  slate  of  the  task 
force  Qficl  the  Cm*  machine  as  a whoio 
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Sumiriary 

This  paper  represents  a status  report  on  the  design  of 
the  firmware  and  software  for  mfinagcment  of  a distributed 
nuj!ti}>rocossor  colled  Cm*  and  tlie  software  construction 
philosophy  which  mfluenccd  its  dcs’gn.  Wo  have  described 
the  lowest  levels  of  the  kernel;  some  of  the  microcode  and 
all  of  the  software  implementing  what  we  have  desenberi 
now  exists. 

Resides  continu'iuj  wiDi  tlie  design  and  implcmor^tation 
of  f^irthcr  levels  of  software,  we  intend  to  experiment  witli 
the  placement  and  execution  of  kernel  code  witlun  different 
Cm*  configurations.  Param.'*ters  of  these  experiments  will 
Include  varying  the  physical  location  of  the  krrtu*l  code,  the 
number  of  copies  of  that  code  os  well  ns  wl>ich  conputor 
modulus  can  execute  different  portions  of  the  code 

Tor  exampio.  one  experiment  is  to  hmil  the  number  of 
processors  that  can  execute  MC-i?  code  to  (say)  two 
proc(!SSors  in  a cluster  If  user  prorjr.ims  cxecutotg  on 
procr'ssors  Other  than  the  dcsinnated  two  request  MC-2 
functions,  tJieir  requests  wiU  bo  rrcoriieO  so  (f)at  fho 
dcsKj'Pited  two  processors  can  process  these  requests  at 
some  later  time.  The  motivation  for  sucli  an  arrangement  is 
that  a processor  is  riiuc»>  more  efficient  if  it  executes  code 
from  its  local  memory. 

In  arJdition  to  such  oporatnu.j  system  cxpormients.  we 
plan  M number  of  experiments  employ. ng  Cm*  in  the  solution 
of  diffnront  types  of  oppllcal.ons  problems. 


A 
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